Fog Computing: IT Compute Stacks meet Open Architecture Control Reply

Fog computing is getting more popular and is breaking ground as a concept for deploying the Industrial IoT. Fog computing is defined by the OpenFog Consortium as “a system-level horizontal architecture that distributes resources and services of computing, storage, control and networking anywhere along the continuum from Cloud to Things.” Looking further into the definition, the purpose is to provide low-latency, near-edge data management and compute resources to support autonomy and contextually-aware, intelligent systems.

In particular, fog computing facilitates an open, internet architecture for peer-to-peer, scalable compute systems that supports edge analytics, local monitoring and control systems. It’s this latter application to control that I think is particularly interesting.  Control systems have been in place for decades across multiple industries, and recently some of these industries have moved to create interoperable, open control architectures. In this blog, we’ll take a look at some existing open architecture frameworks that bear a resemblance to Fog and how Fog computing and these framework initiatives could benefit from cross-pollination.

Open Architecture Control

Back in 2004, the Navy started developing its Navy Open Architecture. In an effort to reduce costs and increase speed and flexibility in system procurement, the DoD pushed industry to establish and use open architectures. The purpose was to make it simpler and cheaper to integrate systems by clearly defining the infrastructure software and electronics that “glue” the subsystems or systems together. The Navy selected DDS as their publish-subscribe standard for moving data in real time across their software backplane (Figure 1 below).

Screen Shot 2017-04-11 at 7.57.41 PM

Figure 1. The Navy Open Architecture functional overview. Distribution and adaptation middleware, at the center, is for integrating distributed software applications.

Fast forward to now and we find the OpenFog Consortium’s reference architecture looks very much like a modern, IT-based version of what the Navy put together back in 2004 for open architecture control. Given that this Navy Open Architecture is deployed and running successfully across multiple ships, we can feel confident that fog computing as an architectural pattern makes sense for real-world systems. Also, we can likely benefit from looking at lessons learned in the development and deployment of the Navy’s architecture.

OpenFMB

The Open Field Message Bus (OpenFMB) is a more recent edge intelligence, distributed control framework standard for smart power-grid applications. It is being developed by the SGIP (Smart Grid Interoperability Panel). Energy utilities are looking at ways to create more efficient and resilient electricity delivery systems that take advantage of clean energy and hi-tech solutions.

Instead of large, centralized power plants burning fossil fuels or using nuclear power to drive spinning turbines and generators, Distributed Energy Resources (DERs) have emerged as greener, local (at the edge of the power grid) alternatives that do not have to transmit electricity over long distances. DERs are typically clean energy solutions (solar, wind, hydro, geothermal) that provide for local generation, storage and consumption of electricity. But DERs are intermittent and need to be managed and controlled locally, as opposed to centrally, which is all that is available in the current power grid.

Distributed intelligence and edge control is the solution. The OpenFMB framework is being deployed and proven in smart grid testbeds and field systems. Looking at the OpenFMB architecture (Figure 2 below), you can see the concept of a software integration bus clearly illustrated.

Screen Shot 2017-04-11 at 7.59.02 PM

Figure 2. OpenFMB architecture integrates subsystems and applications through a central, real-time publish-subscribe bus.

Like the Navy Open Architecture, the OpenFMB distributed intelligence architecture looks very much like a fog computing environment. Since OpenFMB is still under development, I would bet that the OpenFog Consortium and OpenFMB project team would benefit by collaborating.

OpenICE

Patient monitoring, particularly in intensive care units and emergency rooms is a challenging process. There can be well over a dozen devices attached to a patient – and none of them interoperate. To integrate the data needed to make intelligent decisions about the welfare and safety of the patient, someone has to read the front-end of each device and do “sensor fusion” in their head or verbally with another person.

OpenICE, the Open Source Integrated Clinical Environment, was created by the healthcare IT community to provide an open architecture framework that supports medical device interoperability and intelligent medical application development. OpenICE (Figure 3 below) provides a central databus to integrate software applications and medical devices.

openICE data flow RTI DDS

Figure 3. The OpenICE distributed compute architecture, with DDS-based databus, facilitates medical device and software application integration.

Again, the OpenICE architecture supports distributed, local monitoring, integration and control and looks very much like a fog architecture.

And now Open Process Automation

More recently, Exxon-Mobil and other process automation customers have gathered via the Open Process Automation Forum to begin defining an open architecture process automation framework. If you look at the various refineries run by Exxon-Mobil, you’ll find distributed control systems from multiple vendors. Each major provider of process automation systems or distributed control systems has its own protocols, management interfaces and application development ecosystems.

In this walled garden environment, integrating a latest and greatest subsystem, sensor or device is much more challenging. Integration costs are higher, device manufacturers have to support multiple protocols, and software application development has to be targeted at each ecosystem. The opportunity for the Open Process Automation Forum is to develop a single, IIoT based architecture that will foster innovation and streamline integration.

Looking at the Exxon-Mobil diagram below, we find, again, an architecture centered around an integration bus, which they call a real-time service bus. The purpose is to provide an open-architecture software application and device integration bus.

Exxon-Mobile Process Automation Architecture

Figure 4. Exxon-Mobil’s vision of an open process automation architecture centered around a real-time service bus.

Again, we see a very similar architecture to what is being developed in the IIoT as fog computing.

The Opportunity

Each of these open architecture initiatives is looking to apply modern, IIoT techniques, technologies and standards to their particular monitoring, analysis and control challenges. The benefits are fostering innovation with an open ecosystem and streamlining integration with an open architecture.

In each case, a central element of the architecture is a software integration bus (in many cases a DDS databus) that acts as the software backplane facilitating distributed control, monitoring and analysis. Each group is also addressing (or needs to address) the other aspects of fog computing like end-to-end security, system management and & provisioning, distributed data management and other aspects of a functional fog computing architecture. They have the opportunity to take advantage of the other capabilities of Industrial IoT beyond control.

We have the opportunity to learn from each effort, cross-pollinate best practices and develop a common architecture that spans multiple industries and application domains. These architectures all seem to have very similar fog computing requirements to me.

Industrial Internet Connectivity Document Evaluates Core Standards: DDS, OPC-UA, WebServices 3

The Industrial Internet Consortium has released an important part of its Reference Architecture guidance: its Connectivity Framework document. This is actually pretty important; this document dives into the detail on connectivity for IIoT systems, establishes criteria for evaluating connectivity technologies/standards and puts forward some likely technologies for core connectivity standards, including DDS, OPC-UA and WebServices. In other words, there is some really valuable guidance here.

What is Connectivity for IIoT Systems?

According to the Industrial Internet of Things Connectivity document, “connectivity provides the ability to exchange information amongst participants within a functional domain, across functional domains within a system and across systems. The information exchanged may include sensor updates, events, alarms, status changes, commands, and configuration updates.” More concretely, connectivity is the critical, cross-cutting function that supports interoperability within and across IIoT systems. Moving beyond the current mish-mash of proprietary and vertical-industry-specific standards to an open IIoT standards-based framework is the goal of this work.

Looking at Connectivity from a technical viewpoint, figure 1 shows where the Connectivity function lies on a Network, Connectivity, and Information stack, and it divides Connectivity into 2 different layers: Transport and Framework. The Transport layer provides technical interoperability, with “Bits and Bytes shared between endpoints, using an unambiguously defined communication protocol.” The Framework layer provides syntactic interoperability, with “Structured data types shared between endpoints. Introduces a common structure to share data; i.e., a common data structure is shared. On this level, a common protocol is used to exchange data; the structure of the data exchanged is unambiguously defined.” Addressing connectivity needs up through the syntactic interoperability provided by the connectivity framework layer and assessing connectivity framework standards is one of the important contributions of this document.

iicf_connectivity_stack

Figure 1. Connectivity, using the Networking functions below – Internet Protocol, provides the layers for communicating messages and data between system participants.

The Concept of a Core Connectivity Standard.

To ensure interoperability within and across IIoT systems, the Connectivity Framework document recommends the use of a core connectivity standard. Figure 2 shows how this core standard becomes the connectivity bus for the system, integrating native devices and applications directly and legacy, or non-core-standard devices and applications through protocol gateways or bridges. In this way non-standard entities can be “normalized” into the core connectivity standard. This core connectivity reference architecture is central to the IIC’s guidance on ensuring secure, device to device to application interoperability for IIoT systems.

connectivity_ref_arch

Figure 2. Using a Core Connectivity Standard provides for interoperability and streamlined integration within and across IIoT systems.

Evaluating Connectivity Standards.

To reduce the integration and interoperability challenge across different IIoT systems, a key goal of the IIC, the document provides a method and template for evaluating connectivity technologies and standards for the IIoT. It includes assessments of most IIoT standards like DDS, OPC-UA, HTTP/WebServices, OneM2M, MQTT and CoAP. Many of these standards turn out to address different levels of the connectivity stack as you can see in figure 3. Further details on each standard are provided in the document.

iiot_protocol_stds

Figure 3. IIoT connectivity standards and their location on the connectivity stack.

DDS as a Core Connectivity Standard.

From figure 3, you can see that the document assesses 4 connectivity framework standards including DDS. In addition, the Connectivity Framework document provides guidance on requirements for choosing core connectivity framework standards. A core connectivity framework must:

  • Provide syntactic interoperability
    • Provide way to model data, a type-system (e.g. DDS, OPCUA)
    • Can’t be just a “simple” messaging protocol (MQTT, COAP, etc)
  • Be an open Standard with strong governance:
    • from SDOs like IEEE, OASIS, OMG, W3C, IETF
  • Be horizontal & neutral
  • Be stable and deployed in many industries
  • Have standards-defined core gateways to all other connectivity core standards
  • Provide core functions like publish-subscribe, request-reply, discovert, etc.
  • Meet non-functional requirements: performance, scalability, security, …
  • Meet business criteria: not require single components from single vendors, have supported SDKs, have open source implementations, etc.

In figure 4, you can see the 4 potential core connectivity framework standards assessed against these requirements. DDS supports all the requirements and is a promising standard for IIoT systems across all industries.

core_connectivity_table

Figure 4. IIoT Connectivity Core Standards Criteria applied to key connectivity framework standards.

In particular, if you compare DDS with another promising connectivity framework standard, OPC-UA, from figure 5 below, you can see that they address very different system use cases. If your primary challenge is integrating software applications across an IIoT system, then DDS is a good choice. If you challenge is to provide an interface for your edge device to allow system integrators to later integrate it something like a manufacturing workcell, then OPC-UA is a good choice.

system_aspects

Figure 5. Non-overlapping system aspects addressed by the core connectivity framework standards.

As you can see, this IIC document provides a lot of important guidance and clarifying concepts for IIoT connectivity. You can use it’s IIoT connectivity standards assessment profile to assess other standards you may be interested in for your system, or use its guidance to choose among the leading standards. For more detail download the document for yourself.

2nd Version of the Industrial Internet Reference Architecture is Out with Layered Databus Reply

celebration

A year and a half ago the IIC released the first version of the Industrial Internet Reference Architecture (IIRA) – now the second version (v1.8) is out. It includes tweaks, updates and improvements, the most important or interesting of which is a new Layered Databus Architecture Pattern. RTI contributed this new architecture pattern in the Implementation Viewpoint of the IIRA because we’ve seen it deployed by hundreds of organizations that use DDS. Now it’s one of the 3 common implementation patterns called out by the new version of the IIRA.

So, what is a databus? According to the IIC’s Vocabulary document, “a databus is a data-centric information-sharing technology that implements a virtual, global data space, where applications read and update data via a publish-subscribe communications mechanism. Note to entry: key characteristics of a databus are (a) the applications directly interface with the operational data, (b) the databus implementation interprets and selectively filters the data, and (c) the databus implementation imposes rules and manages Quality of Service (QoS) parameters, such as rate, reliability and security of data flow.”

For those who know the DDS standard, this should sound familiar. You can implement a databus with a lower level protocol like MQTT, but DDS provides all the higher-level QoS, data handling, and security mechanisms you will need for a full featured databus.

As we look across the hundreds of IIoT systems DDS users have developed, what emerges is a common architecture pattern with multiple databuses layered by communication QoS and data model needs. As we see in the figure below, we’ll usually see databuses implemented at the edge in the smart machines or lowest level subsystems like a turbine, a car, an oil rig or a hospital room. Then above those, we’ll see one or more databuses that integrate these smart machines or subsystems, facilitating data communications between them and with the higher level control center or backend systems. The backend or control center layer might be the highest layer databus we see in the system, but there can be more than these three layers. It’s in the control center (which could be the cloud) layer that we see the data historians, user interfaces, high-level analytics and other top-level applications. From this layer, it’s straightforward to zero in on a particular data publication at any layer of the system as needed. It’s from this highest layer that we usually see integration with business and IT systems.

iira

The Layered Databus Architecture Pattern: one of three implementation patterns in the newly released Industrial Internet Reference Architecture v1.8.

Why use a layered databus architecture? As the new IIRA says, you get these benefits:

  • Fast device-to-device integration – with delivery times in milliseconds or microseconds
  • Automatic data and application discovery – within and between databuses
  • Scalable integration – comprising hundreds of thousands of machines, sensors and actuators
  • Natural redundancy – allowing extreme availability and resilience
  • Hierarchical subsystem isolation – enabling development of complex system designs

If you want to dig into the databus concept, especially as it compares with a database (similar data-centric patterns for integrating distributed systems, but different in the way they integrate via data), take a look at this earlier blog post on databus versus database.

In addition to the new IIRA release, the IIC is getting ready to release an important document on the Connectivity Framework for its reference architecture. Look for much more detail on this document that sets out core connectivity standards for the Industrial Internet.