Using Virtual Parameters for Data Conditioning
Data conditioning is a common task for the OnPing team. OnPing is a real time system, and our customers expect real-time conditioning of incoming values based on context from historical or adjacent data. We can approach data conditioning in many ways, but for pure power, virtual parameters are hard to beat.
Uneven Data Flow
Recently, we were ingesting data from a SCADA platform. The data was sent by exception with disparate time stamps. Something like this:
Notice, as new data arrive the value indicated here increases (as indicated by the thicker line). The line thins out and flattens as data stop flowing in.
This is a common problem encountered int he conditioning process. One nice solution is called data mapping. Virtual parameters can be used to create a map, like this:
The Script is very short. Simply:
latestValueBefore ?now input0
Breaking Down the Script
input0: This assigns the parameter we want to map in the virtual parameter engine.
?now: This represents the time the script is ran. When ran in a historical context, it will adapt as you go.
latestValueBefore: This function pulls the value closest to and before the time indicated by ‘?now time’.
When you apply that parameter to the trend we showed above the gaps are filled:
Accounting for Comm Fails
But, what if the gap is because of a comm fail?
In the case of this parameter – which is a running total – we relied on some prior knowledge to address cases of comm failure. We knew the ‘total’ value was dependent on a rate. In this case, we assumed a zero-rate meant that no accumulation should be happening. So, if the last rate was not 0 and the accumulation was still not updating, then a problem had occured.
More Robust Data Conditioning
There are times where a simple data map doesn’t provide enough fine detail for the task at hand. OnPing’s inferno tools can handle more intricate conditionings as well.
let flowRate = (Option.reduce id 0.0 (latestValueBefore ?now flowParam)) in let optionalVolume = (latestValueBefore ?now volumeParam) in let zero = (Option.reduce id 0.5 (latestValueBefore ?now zeroParam)) in let checkForRecentVolume = fun i -> valueAt i ?now in if (flowRate < zero) then optionalVolume else checkForRecentVolume volumeParam
Breaking Down a Longer Script
The inferno code sample above is a bit longer than the first. Several key variables can help us identify what is happening.
- flowRate: This is the parameter we are conditioning our output on.
- optionalVolume: This is one of the 2 values that might end up being the output.
- zero: We are dealing with analog numbers so we define zero explicitely.
'flowRate', we use the
reduce function. These values are all coming from data streams. The
zero data stream respectively. When you pick a value out of this stream using
valueAt, you have to handle the fact that the returned value is
Optional. It may have
(Some val) or
In the case of these two, use the Option.reduce function to deal with the ‘None’ case. The pattern
(Option.reduce id (defaultValue) (latestValueBefore …) is very common in inferno scripts. It says return the value you find in
latestValueBefore or return a
defaultValue. The default values in the script above are 0.0 for
flowRate and 0.5 for zero.
The script checks if the
flowRate is less than the
zero. If so,
optionalVolume is returned untouched. Otherwise, the script uses the
checkForRecentVolume function to find
'valueAt' for the current time and the volume. If no value is found, a comm failure
Visualizing the Script Flow
At the end of this process, we are left with a very concise representation of possibilities. Below is a simplified flow chart of the above script.
Now we have data that is respectful of error (instead of pretending all is always well) but still fills in the gaps as needed!
Hopefully this example provides an idea of one way virtual parameters can be used to condition data in various interesting contexts. Contact us at email@example.com to let us know when you have something interesting to use this for, and we’d love to assist in your project!