A Detailed Overview of OnPing

OnPing is a suite of data and device management systems designed to make interacting with industrial machine data simpler. This document is an attempt to walk through all the various systems involved in OnPing one by one.

There are lots of capabilities and interacting parts involved in OnPing and it can be overwhelming. However, having everything written down in one place can be a good way of finding one part you understand and using it to understand others.

This document is intended to reveal the relationship between systems in OnPing. It serves a secondary purpose as a reference for people trying to architect similar systems. There are a lot of moving parts!

The Lumberjack Application System (LAS) is an application manager designed to run on top of a Linux build. It contains drivers for an extensive set of industrial devices including those with manufacturer specific protocols.

OnPing’s systems access these devices directly and send data to and from them easily. LAS is designed to run on very small devices, as small as a Raspberry Pi or ODROID device.

Every running application on the LAS is completely managed. The data it contains is backed up. The network settings it has are remotely configurable. The system makes every effort to minimize the amount of data sent over the wire back to its sources. A Permissions and Identity system are both stored locally and sync’d with cloud devices.

OnPing’s Lumberjack Application System takes security seriously: all data is encrypted and all transmission is client initiated – eliminating the need for static IPs that so often become vectors for attack. Permissions on the network can be revoked remotely.

All this attention to the networking and security settings upfront is essential to the design of everything else in OnPing. The goal is to create a system that handles these issues at the ground level so that everywhere else you can just talk about Lumberjack A talking to Lumberjack B and that is it.

The Data Management systems encompass the ways we store historical and configuration data.

Data flows from the Lumberjacks to the data management system. The important parts are:

• RTU Client – Lumberjack application responsible for interacting with various drivers and applications running on the lumberjack.
• RTU Manager – Server side application responsible for gathering data from lumberjacks and feeding it to cloud applications.
• TachDB (time series database) Reads data from field devices and stores in a multi-resolution format for quick display.
• TachDB Recovery – System where data is recovered as chunks in the event of comm loss.

Currently much of the visualization system resides in the OnPing binary directly, but as each part is asked to do more this will continue to evolve – visualizations included.

There are two items in OnPing that require identity management.

1. The users of OnPing, this is primarily handled through the plowtech auth server.
2. The lumberjacks in OnPing, this is handled through the lumberjack identity server.

Every User in OnPing has an identity based on their email address. They also have a set of groups they are members of that provide permissions for them. While different than a more common roles based authorization system, it shares many features with this approach.

The alarm system in OnPing is split across a few different services. Alarming is a very complicated activity, but its ubiquity in automation systems often causes people to forget just how difficult it is.

A proper system must be highly distributed, highly available and dependably consistent – those things don’t mix well!

AlarmDB is responsible for combining all the components of an alarm into an unified alarm structure – ready to test and run. The tasks involved include:

• • Gathering the rules about whether an alarm is in a ‘Tripped’ state 
  • Gathering whether an alarm has been ‘Enabled’ or ‘Disabled’ 
  • Gathering the status of the ‘Alert’ options currently selected by users
  • Combining all this information into a single presentation to be distributed.

Once this has been done it is sent to an Alarm State Server through an alarm coordinator. Alarm Coordinator presents Alarm DB with routing information. From there, the service recognizes how and where to send the data.

Alarm State Server does the state machine work and the callout work in the system. Currently, it is split into dozens of nodes, each responsible for 1000s of alarms. Alarms depend on a lot of timing rules that can be quite complicated. Once you know an alarm has tripped it still must be determined:

• • How long has the alarm been tripped? 
  • How long until a tripped alarm should call out?
  • When alarm has called someone, how long should it wait to call again? 
  • In case of a phone or other callout system error , how long to wait to try again? 
  • Should clear calls happen? 
  • What is the state of the Alarm? This must be checked simultaneously with the above.

Lastly, we need to Log what has happened:

• Who was alerted?
• How were they alerted?
• When were they alerted?

Virtual parameters are used by alarms and visualization alike. They provide on demand calculation using OnPing-mask-script to do the calculating. They are a distributed application in much the same way our Alarm system is.

We are often asked what is required to stand up highly configurable SCADA systems. Here, we wanted to create an outline of the major systems involved making OnPing work. The development of each system required a huge amount of effort and care to architect according to the greater vision of a more simple, secure and easy to use automation platform. Together they compose a unique and integrated system for managing data and devices for industrial applications.

The main lesson we have learned is that managing complexity is the key to keeping these systems understandable and maintainable. We have designed the OnPing systems to scale well and have a high degree of flexibility to absorb new technology and features into the system. 

This overview of primary OnPing technical infrastructure is meant to distill the essence of OnPing into a simple, functional model of the platform. 

Fundamentally, OnPing aims to reduce complexity and make data make sense. Simplifying complicated technical processes in system orchestration enables us to focus on what matters most – building better automation systems.