Using Virtual Parameters for Data Conditioning
Data conditioning is a common task for the OnPing team. OnPing is a real time system, and our customers expect real-time conditioning of incoming values based on context from historical or adjacent data. We can approach data conditioning in many ways, but for pure power, virtual parameters are hard to beat.
Uneven Data Flow
Recently, we were ingesting data from a SCADA platform. The data was sent by exception with disparate time stamps. Something like this:
Notice, as new data arrive the value indicated here increases (as indicated by the thicker line). The line thins out and flattens as data stop flowing in.
Data Mapping
This is a common problem encountered int he conditioning process. One nice solution is called data mapping. Virtual parameters can be used to create a map, like this:
The Script is very short. Simply:
latestValueBefore ?now input0
Breaking Down the Script
input0: This assigns the parameter we want to map in the virtual parameter engine.
?now: This represents the time the script is ran. When ran in a historical context, it will adapt as you go.
latestValueBefore: This function pulls the value closest to and before the time indicated by ‘?now time’.
When you apply that parameter to the trend we showed above the gaps are filled:
Accounting for Comm Fails
But, what if the gap is because of a comm fail?
In the case of this parameter – which is a running total – we relied on some prior knowledge to address cases of comm failure. We knew the ‘total’ value was dependent on a rate. In this case, we assumed a zero-rate meant that no accumulation should be happening. So, if the last rate was not 0 and the accumulation was still not updating, then a problem had occured.
More Robust Data Conditioning
There are times where a simple data map doesn’t provide enough fine detail for the task at hand. OnPing’s inferno tools can handle more intricate conditionings as well.
Example:
let flowRate = (Option.reduce id 0.0 (latestValueBefore ?now flowParam))
in let optionalVolume = (latestValueBefore ?now volumeParam)
in let zero = (Option.reduce id 0.5 (latestValueBefore ?now zeroParam))
in let checkForRecentVolume = fun i -> valueAt i ?now
in if (flowRate < zero)
then optionalVolume
else checkForRecentVolume volumeParam
Breaking Down a Longer Script
The inferno code sample above is a bit longer than the first. Several key variables can help us identify what is happening.
- flowRate: This is the parameter we are conditioning our output on.
- optionalVolume: This is one of the 2 values that might end up being the output.
- zero: We are dealing with analog numbers so we define zero explicitely.
For the 'zero'
and 'flowRate'
, we use the reduce
function. These values are all coming from data streams. The flowParam
and zero
data stream respectively. When you pick a value out of this stream using latestValueBefore
or valueAt
, you have to handle the fact that the returned value is Optional.
It may have (Some val)
or None
.
In the case of these two, use the Option.reduce function to deal with the ‘None’ case. The pattern (Option.reduce id (defaultValue) (latestValueBefore …)
is very common in inferno scripts. It says return the value you find in latestValueBefore
or return a defaultValue
. The default values in the script above are 0.0 for flowRate
and 0.5 for zero.
The script checks if the flowRate
is less than the zero
. If so, optionalVolume
is returned untouched. Otherwise, the script uses the checkForRecentVolume
function to find 'valueAt'
for the current time and the volume. If no value is found, a comm failure occured
.
Visualizing the Script Flow
At the end of this process, we are left with a very concise representation of possibilities. Below is a simplified flow chart of the above script.
Now we have data that is respectful of error (instead of pretending all is always well) but still fills in the gaps as needed!
Contact Us
Hopefully this example provides an idea of one way virtual parameters can be used to condition data in various interesting contexts. Contact us at info@onping.net to let us know when you have something interesting to use this for, and we’d love to assist in your project!