Control File

Data Management Interface (DMI) : Control File-Executable DMI : Control File

Control File

The control file is an ASCII file which the user provides. The purpose of the control file is as follows:

• Map between object.slots in the Workspace and data files. In the case of an import DMI, data is imported from the data files to the object.slots; in the case of an export DMI, data is exported from the object.slots to the data files.

• Provide information about the data files.

Each control file entry contains three pieces of information:

• An object.slot specification.

• One or more file names listed as file = name

• Optional keyword = value pairs.

The control file format is:

object.slot: file=name keyword=value keyword=value ...

If a file name contains spaces then it must be enclosed in double quotations marks (“), for example:

object.slot: file=”file name” keyword=value keyword=value ...

The DMI must also be configured to allow spaces in file paths; see Allow Spaces in File Paths for details.

Object.Slot Specifications

The object.slot specifications indicate which slots will import from or export to data files. There are four ways of specifying the object:

• <object name> - A specific object name, e.g., “FtLoudoun,” which matches the object with that name. Aggregate object elements are specified as <aggregate name>:<element name>.

• <object type> - An object type, e.g., “LevelPowerReservoir,” which matches all objects of that type.

• <sub-basin name> - A sub-basin name, which matches all objects in the sub-basin.

• * - A wild card which matches all objects of all types.

There are three ways of specifying the slot:

• <slot name> - A specific slot name, e.g., “Inflow,” which matches the slot with that name.

• <slot type> - A slot type, e.g., “SeriesSlot,” which matches all slots of that type.

• dispatch - Any dispatching slots.

There are two ways of specifying a column within multi-column slots:

• <column name> - A specific column name, which matches one of the columns in the selected slot.

• no name - If no name is given, the DMI defaults to the first column.

An account or supply can be specified like so:

• <object name>^<account name>

• <supply name>.Supply - the “.Supply” portion of the string is necessary to tell RiverWare that this is a supply in the accounting system and not a slot in the physical system. If a DMI is used to bring in values, it should be setting supplies in the accounting system rather than accounting slots. See Setting Slots Versus Setting Supplies in Accounting for details.

Also, the slots in a Slot Set can be specified using the SlotSet keyword, as follows. See Slot Sets in User Interface) for details.

SlotSet.<slot set name>

For example, SlotSet.Reservoir Inflows would expand to all of the slots in the Reservoir Inflows set.

The above slot specifications lead to many ways of specifying the object.slot pairs, as follows:

<sub-basin name>.<slot name>.<column name>

<sub-basin name>.<slot name>

<sub-basin name>.<slot type>.<column name>

<sub-basin name>.<slot type>

<sub-basin name>.dispatch.<column name>

<sub-basin name>.dispatch

*.<slot name>.<column name>

*.<slot name>

*.<slot type>.<column name>

*.<slot type>

*.dispatch.<column name>

*.dispatch

<supply name>.Supply

SlotSet.<Slot Set Name>

These pairs are listed from most exact to least exact. It is possible that more than one control file entry will select a specific object.slot pair in the Workspace. In this case, the more exact entry will always take precedence. For example, in the control file:

# All Confluence objects, Inflow1 slot.

Confluence.Inflow1: file=~/%o.inflow1.dat

# Confluence01 object, Inflow1 slot.

Confluence01.Inflow1: file=~/confluence01.inflow1.dat

Both entries will select Confluence01's Inflow1 slot (the first through an object type, the second through an object name). The more exact entry, the second entry, will take precedence.

There are two other concepts this example introduces: the control file may contain comments (lines beginning with #) and the file name may contain special characters (~ and %), described below.

Finally, note that object and slot names and types must be specified exactly as they appear in the user interface, including case and whitespace.

File Name and Directory Specification

The file name indicates from which data files the object.slots will be imported or to which data files the objects.slots will be exported. The file name may be either relative (beginning with ~/) or absolute (beginning with C:/). If the file name is relative, then it is relative to the DMI's working directory. The DMI's working directory is where it creates its temporary files. If the RIVERWARE_DMI_DIR environment variable is set, the DMI's working directory is $RIVERWARE_DMI_DIR/dmi_name; otherwise, the DMI's working directory is /tmp/dmi_name. In both cases, dmi_name is the name that appears in the DMI Selector list.

The file name may contain the following directives (currently, it may contain at most one of each):

• %o - The %o is replaced with the object name. These are useful when using symbolic references to object in the file specifications.

• %s - The %s is replaced with the slot name. These are useful when using symbolic references to slots in the file specifications.

• %t - The %t is replaced with the current MRM trace number. If there is no trace number because the DMI is not executed as part of MRM, a trace of 1 is assumed.

• %tempdir - The %tempdir is replaced by a guaranteed unique temporary directory as described below.

Note: Because dmi names, object names and slot names are mapped to directory and file names, characters which are not valid in directory and file names (currently all characters other than A-Za-z0-9/_.) are replaced with _ when the directory or file name is created.

Consider the following example:

RIVERWARE_DMI_DIR is C:\temp\local\riverware

The DMI name is “test dmi”

The control file contains the following entry:

Canal.Flow 1: file=~\%o%t.%s.dat

The file which the “Tellico Canal.Flow 1" object.slot for trace 23 or an MRM would import from or export to would be:

C:\temp\local\riverware\test_dmi\Tellico_Canal23.Flow_1.dat

The ~ has been replaced with the value of the working directory, RIVERWARE_DMI_DIR\<dmi name>, %o has been replaced with the object name, %t has been replaced with the trace number, and %s has been replaced with the slot name (with the spaces having been replaced with _).

Multiple files can be specified on a single control file line entry for input dmi’s. This enables the user to import data into a slot from two or more files. Use of the data_date file keyword allows the user to specify where in the series slot each file should be placed. The syntax for multiple files is

Object.Slot: file=name file=name

The data files are imported from left to right. If the import = resize is specified, the slot will grow to accommodate the data. Overlapping data is allowed but values will be overwritten by the later data files. If the incoming data does not overlap and there are gaps, any untouched values that existed on the slot will remain.

Another control file directive available is %tempdir, which is replaced by a guaranteed unique temporary directory. When the DMI is invoked RiverWare generates the directory name, makes the directory and passes the directory to the DMI executable via the RW_DMI_TEMPDIR environment variable. The executable should be able to recognize RW_DMI_TEMPDIR. After the DMI finishes RiverWare removes the directory. The directory name will be similar to the following:

b05741ad-2b57-4f82-8044-bda67606e4c9

Because RiverWare removes the directory you most likely won't see these directories, but if you kill RiverWare or RiverWare crashes a directory might be left behind, in which case you can safely remove it.

Time Range Specification

The time range of data that is imported or exported by a Control File-Executable DMI is controlled in the control files and possibly in the data file. If not specified, the default is to export/import the range of the slot. But if you want to limit the data that is imported or exported use one or more of the following options:

• For an output DMI, you can limit the output to only the run range using the “Limit Output to Run Range” check box described in Limit Output to Run Range. See also the note below.

• For an input DMI, specify the timestep of the data that should be imported in the data file using the data_date keyword. This specification is in the data file itself so it is on a per-slot basis. See the description of data_date in Data Files.

• For either an input or output DMI, specify the start_timestep and end_timestep in the control file for each slot specification. Both of these keywords allow you to also specify the timestep symbolically using RPL syntax. For example, the following are valid entries:

– start_timestep=“24:00 November Max DayOfMonth, 1996”

– end_timestep=“Start Month Start DayOfMonth, Start Year - 2 days”

– start_timestep=“Current Timestep” (Current timestep is only supported for rule executed DMIs as they know at which timestep the DMI is executing, see Executing DMIs From Blocks in RiverWare Policy Language (RPL).

– end_timestep=“24:00 3/1/2021”

See DATETIME in RiverWare Policy Language (RPL) for details on datetimes. Basically, any fully specified datetime can be used. No @ symbol is necessary but quotes are necessary when specifying datetimes that contain spaces.

When using these keywords on an input DMI, the data file does not have to have the same start_date and end_date as the control file. The data file should encompass the desired data. Otherwise a warning message will be issued, and no data will be imported. If the data file does not have a start_date keyword, the data will be imported starting with the first value in the data file imported to the start_timestep. Also, using the data_date keyword in the data file is possible. It is recommended to create data files that match the control file’s start_timestep and end_timestep for improved performance.

For an input DMI, the keyword end_timestep will not grow a slot. Use the import=resize to grow a slot to fit the imported data. (See import=<fixed>|<resize>.)

Note: The time range of the slot after import will be based on the start_date and end_date keywords in the data file consistent with the description for import=<fixed>|<resize>. Data will be imported only for the range specified by the start_timestep and end_timestep in the control file, and any additional timesteps will contain NaN.

On an output DMI, if you specify start_timestep and end_timestep (a symbolic range) in the control file and Limit Output to Run Range in the DMI interface, the following applies: If the run range and symbolic range are disjoint, constrain the data transfer to the symbolic range. Otherwise, constrain the data transfer to the intersection of the run range and symbolic range.

Optional keyword=value Pairs

The optional keyword=value pairs provide additional information about the data files the slots will import. (The one exception to this is the file keyword, which identifies the data file itself.) Although the keywords are not case sensitive, the values can be.

The currently supported keywords are:

file=<data file>

The data file the slot will import from or export to, as described above in File Name and Directory Specification.

start_timestep=<datetime>

An optional datetime that indicates the timestep at which to start importing/exporting, as described above in Time Range Specification. If not specified for an input DMI and the data file has a start_date keyword, the data are imported beginning with the data file’s start_date. If not specified for an input DMI and the data file does not have a start_date, the slot’s start timestep is used. If not specified for an output DMI, the slot’s start timestep is used. The datetime can be specified symbolically using RPL syntax.

end_timestep=<datetime>

An optional datetime that indicates the timestep at which to end the importing/exporting, as described above in Time Range Specification. If not specified or an input DMI, the end of the import range is based on the combination of the end_date keyword in the data file and the import=<fixed>|<resize> keyword in the control file as described in import=<fixed>|<resize>. If not specified for an output DMI, the end timestep of the slot is used. The datetime can be specified symbolically using RPL syntax.

units=<units>

The units of the data the slot will import or export. <units> must match exactly one of the units defined in the Units List dialog (see Display Units in User Interface for details on the Units List).

Note: If units are not specified in the control file, then it defaults to the current setting in the user interface as specified by the active Unit Scheme; (see Unit Schemes in User Interface for details). Since the unit scheme can change from invocation to invocation, it is strongly recommended that the control file always contain units.

scale=<scale> (float)

The scale of the data the slot will import or export.

Note: If scale is not specified in the control file, then it defaults to the current setting in the user interface as specified by the active Unit Scheme; (see Unit Schemes in User Interface for details). Since the unit scheme can change from invocation to invocation, it is strongly recommended that the control file always contain the scale (even if the scale is 1.0).

precision=<display>|<model>

The precision of the data the slot will import or export. The <display> precision is that specified in the Unit Scheme for the slot. The <model> precision is 17, the same precision as all internal calculations. The default is <display>.

import=<fixed>|<resize>

This keyword controls how the DMI sets the time series range of the series slot. The default is fixed.

For either import option, the Begin Timestep of the slot is set in the same manner. If the data file contains a start_date header, the slot Begin Timestep is set to the start_date from the data file. If the data file does not contain a start_date header, the slot maintains its existing Begin Timestep.The data import always starts on the new Begin Timestep of the slot with the first value in the data file.

For import=resize, if the data file contains an end_date header, the End Timestep of the time series range of the series slot will be set to the later of the end_date header in the data file or the number of timesteps necessary hold the number of data values in the data file given the updated Begin Timestep of the slot from above. If the data file does not contain an end_date header, the End Timestep of the time series range of the slot will be extended, if necessary, to hold the number of data values in the data file given the updated Begin Timestep of the slot from above. The End Timestep of the slot will never be moved earlier if there is no end_date header in the data file.

For import=fixed, if the data file contains an end_date header, the End Timestep of the series slot is set to the end_date from the data file. Data will not be imported past the new End Timestep of the slot. If the data file does not contain an end_date header and the existing End Timestep on the slot is later than the updated Begin Timestep, the slot maintains its existing End Timestep. Data will not be imported past the End Timestep of the slot. If the existing End Timestep on the slot is equal to or earlier than the updated Begin Timestep, then the slot will be expanded to hold all of the data values in the data file starting with the updated Begin Timestep.

Note: If the control file uses the start_timestep and end_timestep keywords, the time range for the slot after import will still be as described here, but data will only be imported for the time range corresponding to the start_timestep and end_timestep in the control file. Any additional timesteps will contain NaN. (See Time Range Specification.)

flags=<true>|<false>

The flags keyword specifies whether or not Series Slot flag values (Input, Target, Best Efficiency, Max Capacity, Drift, and Unit Values) will be exported or imported with the data. The syntax is flags=true; the default is <false>. If an input DMI, the keyword pairs in the data file should be specified as “value flag” with the flag being an integer value, e.g. Input = 1, Output = 2, etc.

aggregate=<true>|<false>

The aggregate keyword is used to specify whether or not the DMI should import/export all time series slots on an Aggregate Series slot or just the first time series column. If present, with the value “true”, and if the slot indicated is an Aggregate Series slot, the data file is expected to (on import) or will (on export) contain data for all subslots (columns) in the Aggregate Series slot. If the keyword=value pair is absent, or if its value is “false”, only the data for the first subslot (column) of the Aggregate Series slot are present. Table 2.1 shows the resulting metacontrol file entry for the range of possible control file entries and , the aggregate keyword values

Table 2.1 Possible entries when using the aggregate keyword.
Control File entry	aggregate keyword	Metacontrol result	Comment
Obj.Slot	false	Obj.Slot.ColumnLabel	Only the first column of the agg is import/exported
Obj.Slot	true	Obj.Slot	All columns are imported/exported
Obj.Slot.ColumnLabel	false	Obj.Slot.ColumnLabel	Only the specified column is imported/exported
Obj.Slot.ColumnLabel	true	Obj.Slot	A warning is posted that the keyword is inconsistent and will be ignored.

When Aggregate Series slot data are imported or exported in total, the data are in row-major format, one timestep per line. The subslots of the aggregation must be homogeneous in start-date, end-date and step size, and all subslots (columns) must be represented on each line.

If used with the ‘flags=true’ keyword-value pair, the data will appear in the form:

value flag value flag value flag...

The flag is an integer number representing the flag. If the named slot is in fact the first subslot (column) of an Aggregate Series slot, the behavior is the same as if the subslot (column) name were absent.

slot_anno = <true>|<false>

The slot_anno keyword indicates whether to import or export series slot notes; see Exporting Notes Using DMIs in User Interfacefor details. The default value for slot_anno is false, meaning no Notes will be written.

Note: The slot_anno keyword cannot be used in combination with the slot_set_info = <true>|<false> or slot_opt_sol_info = <true>|<false> keywords.

slot_set_info = <true>|<false>

The slot_set_info keyword indicates whether to export series slot value information. The default value for slot_set_info is false, meaning no information will be written.When true, the information corresponds to the tool tips seen on slot values on slot and SCT dialogs () will be written to the file; see Series Slots in User Interface for details. The string appears after the value. Examples of the information are “Set by DMI: Input DMI", “Set by Rule: (7) Flood Control”.

Note: The slot_set_info keyword cannot be used in combination with the slot_anno = <true>|<false> or slot_opt_sol_info = <true>|<false> keywords.

slot_opt_sol_info = <true>|<false>

The slot_opt_sol_info keyword indicates whether to export series slot optimization solution information . The default value for slot_opt_sol_info is false, meaning no information will be written. When true, the available per slot/value optimization solution information will be written to the file. The string appears after each value. Examples of the information are “Frozen at Lower Bound”, or “Frozen by (3) Minimum Load 47.6% between limits set by 3.1.1.1 and 2.1.1.1.”. See Tooltips on Variables in Optimization for details.

Note: The slot_opt_sol_info keyword cannot be used in combination with the slot_anno = <true>|<false> or slot_set_info = <true>|<false> keywords.

User keyword=value Pairs

The user keyword=value pairs provide an additional means by which the user can pass parameters to the DMI executable. The user keyword=value pairs are specified in the control file as:

!keyword=value

The keyword=value pairs are written to the metaControl file, with the ‘!’ removed, on all lines they apply to. The DMI executable can then use the keyword=value pairs to influence its behavior.

Example 2.1

# All LevelPowerReservoir Outflows

LevelPowerReservoir.Outflow: file=~/%o.Outflow units=cfs scale=1.0

# All LevelPowerReservoir Pool Elevations,

# but for Hiwassee use a different scale.

LevelPowerReservoir.Pool Elevation: file=~/%o.%s units=ft scale=1.0

Hiwassee.Pool Elevation: file=~/%o.%s units=ft scale=10.0

# All LevelPowerReservoir Storages.

LevelPowerReservoir.Storage: file=~/%o.%s units=cfs-day scale=1.0

# All LevelPowerReservoir Hydrologic Inflows.

LevelPowerReservoir.Hydrologic Inflow: file=~/%o.%s units=cfs scale=1.0

# Finally, all SeriesSlots which are not selected by one of

# the preceding lines. (Remember that *.<slot type> is the

# least exact specification.) The file ~/start_end is a data

# file which contains only the start_date: and end_date:

# attributes. This will serve to change the start date and

# end date for all series slots which import this file.

*.SeriesSlot: file=~/start_end

Revised: 07/05/2022