Using Virtual Parameters for Data Conditioning

Data conditioning is a common task for the OnPing team. OnPing is a real time system, and our customers expect real-time conditioning of incoming values based on context from historical or adjacent data. We can approach data conditioning in many ways, but for pure power, virtual parameters are hard to beat.

Uneven Data Flow

Recently, we were ingesting data from a SCADA platform. The data was sent by exception with disparate time stamps. Something like this: 

Notice, as new data arrive the value indicated here increases (as indicated by the thicker line). The line thins out and flattens as data stop flowing in.

Data Mapping

This is a common problem encountered int he conditioning process. One nice solution is called data mapping. Virtual parameters can be used to create a map, like this:

The Script is very short. Simply:

latestValueBefore ?now input0 

Breaking Down the Script

input0: This assigns the parameter we want to map in the virtual parameter engine.

?now: This represents the time the script is ran. When ran in a historical context, it will adapt as you go. 

latestValueBefore: This function pulls the value closest to and before the time indicated by ‘?now time’. 

When you apply that parameter to the trend we showed above the gaps are filled:

Accounting for Comm Fails

But, what if the gap is because of a comm fail?

In the case of this parameter – which is a running total – we relied on some prior knowledge to address cases of comm failure. We knew the ‘total’ value was dependent on a rate. In this case, we assumed a zero-rate meant that no accumulation should be happening. So, if the last rate was not 0 and the accumulation was still not updating, then a problem had occured. 

More Robust Data Conditioning

There are times where a simple data map doesn’t provide enough fine detail for the task at hand. OnPing’s inferno tools can handle more intricate conditionings as well.

Example:

let flowRate = (Option.reduce id 0.0 (latestValueBefore ?now flowParam))
in let optionalVolume = (latestValueBefore ?now volumeParam)
in let zero = (Option.reduce id 0.5 (latestValueBefore ?now zeroParam))
in let checkForRecentVolume = fun i -> valueAt i ?now
in if (flowRate < zero)
 then optionalVolume
 else checkForRecentVolume volumeParam

Breaking Down a Longer Script

The inferno code sample above is a bit longer than the first. Several key variables can help us identify what is happening.

• flowRate: This is the parameter we are conditioning our output on. 
• optionalVolume: This is one of the 2 values that might end up being the output. 
• zero: We are dealing with analog numbers so we define zero explicitely.

For  the 'zero' and 'flowRate', we use the reduce function. These values are all coming from data streams. The flowParam and zero data stream respectively. When you pick a value out of this stream using latestValueBefore or valueAt, you have to handle the fact that the returned value is Optional. It may have (Some val) or None. 

In the case of these two, use the Option.reduce function to deal with the ‘None’ case.  The pattern (Option.reduce id (defaultValue) (latestValueBefore …) is very common in inferno scripts. It says return the value you find in latestValueBefore or return a defaultValue. The default values in the script above are 0.0 for flowRate and 0.5 for zero. 

The script checks if the flowRate is less than the zero. If so, optionalVolume is returned untouched. Otherwise, the script uses the checkForRecentVolume function to find 'valueAt' for the current time and the volume. If no value is found, a comm failure occured. 

Visualizing the Script Flow

At the end of this process, we are left with a very concise representation of possibilities. Below is a simplified flow chart of the above script.

Now we have data that is respectful of error (instead of pretending all is always well) but still fills in the gaps as needed!

Contact Us

Hopefully this example provides an idea of one way virtual parameters can be used to condition data in various interesting contexts. Contact us at info@onping.net to let us know when you have something interesting to use this for, and we’d love to assist in your project!