OpenFMB cartoon

Speed Your Time to Market with Connext Pro Tools 3

It was two weeks until the demo.

We had this single opportunity to build a working microgrid control system that needed to:

  • Run on Intel and ARM processors
  • Target Linux and Windows platforms
  • Include applications written in C, C++, Java, SCALA, Lua, and LabVIEW
  • Talk to legacy equipment speaking ModBus and DNP3 protocols
  • Perform real-time control while meeting all of the above requirements

In this post, I’ll talk about the real-world problems we faced and how the tools included in RTI Connext DDS Pro helped us solve our integration problems in just a couple of days. Common issues encountered in most projects are highlighted, with specific RTI tools for addressing each. Along the way you’ll find links to supporting videos and articles for those who want a deeper dive. My hope is that you find this a useful jumping off point for learning how to apply RTI tools to make your DDS development quicker and easier.

The Big Demo

This was the first working demo of the Smart Grid Interoperability Panel’s Open Field Message Bus (OpenFMB), a new way of controlling devices at the edge of the power grid in real time by applying IoT technologies like DDS (see this link for more info).

OpenFMB cartoon

Here’s a block diagram of the system showing hardware architectures, operating systems, and languages:

OpenFMB demo net diagram-2

As we brought the individual participants onto the network, we encountered a number of problems. A description of challenges and the tools we used to address each follows. Scan the list of headings and see if you’ve had to debug any of these issues in your DDS system, then check out the links to learn a few new tips.  As you do, think about how you would try to diagnose the problems without the tools mentioned.

Problem: Network configuration problems

Tools: RTI DDS Ping

The team from Oak Ridge National Labs was working on the LabVIEW GUI that would be the main display.  Their laptop could not see data from any of the clients on the network. We checked the basics to make sure their machine was on the same subnet – always check the basics first!  While the standard ping utility can confirm basic reachability between machines, it doesn’t check that the ports necessary for DDS discovery are open.  The rtiddsping utility does exactly that, and it told us in seconds that the firewall installed on their government-issued laptop was preventing DDS discovery traffic.  For a great rundown on how to check the basics, see this community post.

Problem: Is my app sending data?

Tools: Spy, Admin Console

A common question among the vendors using DDS for the first time was whether their application was behaving properly: Was it sending data at the proper intervals, and did the data make sense? For a quick check, we used the RTI DDS Spy utility. Spy provides a simple subscriber that can filter selectively for specific types and topics, and it can print the individual samples it receives, allowing you to quickly see the data your app is writing.  Every vendor used DDS Spy as a sanity check after initially firing up their application.

Sometimes an update to the same topic can come from multiple publishers in the system. Not sure which one wrote the latest update?  A command line switch for Spy (“-showSampleIdentity”) allows you to see where an update originated.

Spy is a console app that can be deployed on embedded targets for basic testing.  Its small size, quick startup, and simplicity are its main advantages.  Details on usage are here.

Problem: Data type mismatch

Tools: Admin Console, Monitor

One vendor reported that in an earlier test they were seeing data from one of the other apps, and now they were not. Admin Console quickly showed us that a data type mismatch was to blame – that is, two topics with the same name but different data types. These types of mismatches can be difficult to diagnose, especially for large types with many members. Admin Console leverages the data-centricity of DDS to introspect the data types as understood by each application in your system. It then presents both a simplified view and an “equivalent IDL” view that makes it easy to compare the types in side-by-side panes. This is especially valuable in situations where you don’t have the source IDL from every application.

In this case, one vendor had not synced up with the GitHub repository for the latest IDL, so they were working from an older version of the file. They pulled the latest files from GitHub, rtiddsgen created new type-specific code for them, and after a quick recompile their app was able to read and write the updated topics.

Data type introspection

Admin Console shows data types

Problem: QoS mismatch

Tools: Admin Console, Monitor

Next to discovery, Quality of Service (QoS) mismatches are the most common problem experienced by DDS users during integration. With so many knobs to turn, how do you make sure that settings are compatible? The OpenFMB project had its fair share of QoS mismatches at first. Admin Console spots these quickly and tells you the specific QoS settings that are in conflict. You can even click on the QoS name and go directly to the documentation. QoS information shared during discovery is used by Admin Console to detect mismatches.

QoS Mismatch

Admin Console identifies a reliability QoS mismatch

Problem: Is the system functioning as expected?

Tools: Admin Console, Monitor

While Spy provides basic text output for live data, you can’t beat a graph for seeing how data changes over time. For more sophisticated data visualization, we turned to Admin Console. The data visualization feature built into Admin Console was a huge help in quickly determining how the system as a whole was working. It even allowed us to scroll through historical data to better understand how we arrived at the current state. To find out more about data visualization, see this short intro video, or this deep dive video.

Data visualization

Visualize your data with Admin Console

Problem: Performance tuning

Tools: Monitor, Admin Console

When it comes to performance tuning, Monitor should be your go-to tool. Monitor works with a special version of the DDS libraries that periodically publish real-time performance data from your application. The debug libraries are minimally intrusive, and the data is collected and presented by Monitor.

Using Monitor, you can learn about:

  • Transmission and reception statistics
  • Missed deadlines
  • High-water marks on caches
  • QoS mismatches
  • Data type conflicts
  • Samples lost or rejected
  • Loss of liveliness

It’s important to note that not every QoS setting is advertised during discovery. Many QoS settings apply to an application’s local resource management and performance tuning, and these are not sent during discovery. With Monitor you can inspect these, too.

Problem: Transforming data in flight

Tools: Prototyper with Lua, DDS Toolkit for LabVIEW

We wanted a large GUI to show what was happening in the microgrid in real time.  The team at Oak Ridge National Labs volunteered to create a GUI in LabVIEW. The DDS Toolkit for Labview allows you to grab data from DDS applications and use it in LabVIEW Virtual Instruments (VIs). There are some limitations however, as we found out. The Toolkit does not handle arrays of sequences, which some types in the OpenFMB data model use. We needed a quick solution that would allow the LabVIEW VI to read these complex data types.

One of the cool new tools in the Connext DDS Pro 5.2 toolbox is Prototyper with Lua. Prototyper allows you to quickly create DDS-enabled apps with little to no programming: define your topics and domain participants in XML, add a simple Lua script, and you can be up on a DDS domain in no time. (Check out Gianpiero’s blog post on Prototyper)

Back at the hotel one evening I wrote a simple Lua script that allows Prototyper to read the complex DDS topics containing arrays of sequences and then republish them to a different, flattened topic for use by the LabVIEW GUI. I was able to test it offline using live data recorded earlier in the lab, which brings us to…

Problem: Disconnected development

Tools: Record, Replay, Prototyper with Lua

A geographically dispersed development team built the OpenFMB demo. With the exception of those few days in Knoxville, no one on the team had access to all the components in the microgrid at one time. So how do you write code for your piece of the puzzle when you don’t have access to the other devices in the system?

When I worked on the Lua bridge for the LabVIEW GUI, I used the Connext Pro Record and Replay services.  In the lab, I had recorded about 10 minutes of live data as we ran the system through all the use cases.  Later that evening in the hotel, I was able to play this data back as I worked on the Lua scripts.  Replay allows you selectively play back topics, looping the playback so it runs continuously.  You can also choose to play the data at an accelerated rate – this is a huge time saver that enables you to simulate days or hours worth of runtime in just a few minutes.

Recording console

Recording Console

One of the really neat things Prototyper does once it’s running is to periodically reload the Lua script.  This made developing the bridge to LabVIEW very quick: Replay played data continuously in an accelerated mode; I had an editor open on the Lua script; and as I made and saved changes they were instantly reflected in Prototyper which was running constantly – no need to restart to see changes to the script.  The conversion script was done in just a couple of hours.

Prototyper also came in handy for quickly creating apps to generate simulated data.  The LabVIEW GUI was developed entirely offline without any of the real-world devices, using some topics generated by the Replay services and others that were bridged or simulated with Prototyper.  I’d email a simulator script to ORNL, they’d do some LabVIEW work and send me an updated VI, and then I’d run that locally to verify it.   ORNL did an amazing job integrating real-time data from the DDS domain along with visual elements from the SGIP cartoons, and the GUI was the centerpiece of the demo.

LabVIEW GUI

The final GUI, written in LabVIEW

Main Takeaways

When we showed up in New Orleans a couple weeks later, the entire system was brought up in about 30 minutes, which is remarkable considering some of the applications (like the LabVIEW GUI) had never even been on a network with the actual hardware. Everything just worked.

The rich set of tools provided by RTI Connext DDS Pro allowed us to solve our integration problems quickly during the short week in Knoxville, and to carry on development at many remote locations. Admin Console, Monitor, DDS Ping, and DDS Spy got our system up and running.  Record, Replay, and Prototyper made it possible for remote development teams to work in the absence of real hardware.  DDS Toolkit for LabVIEW enabled us to create a sophisticated GUI quickly.  And even after the event, we can continue to do development and virtual demos using these tools.

Rapid Data Transformations Are Moments Away! 4

When I started college, everybody was talking about “Information Technology.” At that point I had been programming for quite a while and it wasn’t clear to me what coding had to do with that fancy terminology. After a few more years of coding, I realized the connection: all I do, day in and day out, is move bytes (information) from one memory location to another. Copying the contents of a struct into the socket buffer and sending it out; getting the bytes from the socket buffer and deserializing them into a structure to pass them to the application logic. Well, that’s part of what communication middleware does for you!

RTI Connext DDS implements the OMG DDS Standard. DDS is data-centric middleware. We call those bytes being transferred data, and we assign a type to them. The type describes each byte in your data, so you (and your application) can make good use of them.

Designing a good distributed system means having a good type system in place. Often though, those types carry a lot of information, and it can become pretty difficult to deal with them. Also, you may not need all the information all the time. This often happens when you have heterogeneous systems with nodes that have different capabilities, such as bandwidth constraints or resource limits. In this case, some of the nodes in the system may not be able to handle a whole sample for certain data types, but they still need to be able to receive part of the information.

Let me present a possible scenario first and then suggest one way to solve it.

scenario

Figure 1: Scenario

Let’s say a new testing module for the International Space Station (ISS) has been shipped to space. The module has a device that collects thousands of data points, puts them all in a DDS sample with a specific type, and writes it on the topic, ComplexDataTopic. We will call this device the Emitter.

One of the many different data points this device collects is the temperature from 10 different sensors. It puts all the collected temperature values into a sequence.

enum Unit {
   F,
   C,
   K
};

struct Temp {
    long sensorId;
    double value;
    Unit unit;
}

struct Measurements {
  long deviceId;
  // thousand of more fields. 
  sequence<Temp,10> temperatures;
};

// Open file

The module also comes with a FancyReceiver (what we call a topic subscriber) that runs a UI; it gets all the data contained in each sample and creates statistics and charts so astronauts can easily understand the data. For example, looking that the temperature data, it could calculate the average (mean) or the median, or spot significant differences between sensors.

Let’s now say that back on Earth, NASA scientists need to know what the temperature is on the testing module. They’re not interested in pressure and humidity. Just the temperature. But, since the bandwidth between Earth and the ISS “is what it is”, it’s considered acceptable to receive aggregates of the date. For example, instead of sending all the values for the temperature the testing module will only send the average, which will save the ISS some bandwidth on the link to ground. Basically, they want something that looks like this:

enum Unit {
   F,
   C,
   K
};

struct SimpleMeasurements {
    long deviceId;
    double avgTmp;
    Unit unit;
}

// Open File

We have many options to achieve this result:

  1. We can create an ad-hoc application: it will subscribe to the ComplexDataTopic, get the data we need, create another topic, another type, calculate what we need, and send it over.
  2.  We can use RTI Routing Service: write a custom transformation library and be done with it.
  3.  We can use RTI Connector via the RTI Prototyper with Lua.
  4. We can use RTI Connector via Python or node.js.

I will now explain option 3: Use RTI Prototyper with Lua to easily transform your types. If you are already an RTI Connext user, you will have what you need in your distribution (under the “bin” directory if you are using RTI Connext 5.2, or under the “scripts” directory for older versions). Otherwise, check out this page to learn how to get your free copy of RTI Prototyper.

We will create a new component, the Transformer. The Transformer will subscribe to ComplexDataTopic (with type Measurements) and will Publish to SimpleDataTopic (with type SimpleMeasurements).

The Types

For simplicity, let’s say that the complex type is the following:

<enum name="Unit">
  <enumerator name="F"/>
  <enumerator name="C"/>
  <enumerator name="K"/>
</enum> 
<struct name= "Temp">
  <member name="sensorId" id="0" type="long"/>
  <member name="value" id="1" type="double"/>
  <member name="unit" id="2" type="nonBasic"  
        nonBasicTypeName= "Unit"/>
</struct>
<struct name= "Measurements">
  <member name="deviceId" id="0" type="long"/>
  <member name="humidity" id="1" type="double"/>
  <member name="pressure" id="2" type="double"/>
  <member name="temperatures" sequenceMaxLength="10" id="3" 
        type="nonBasic"  nonBasicTypeName= "Temp"/>
</struct>

<!--Open File -->

The simple type looks like this:

<enum name="Unit">
  <enumerator name="F"/>
  <enumerator name="C"/>
  <enumerator name="K"/>
</enum> 
<struct name= "SimpleMeasurements">
  <member name="deviceId" id="0" type="long"/>
  <member name="avgTmp" id="1" type="double"/>
  <member name="unit" id="2" type="nonBasic"  
        nonBasicTypeName= "Unit"/>
</struct>

<!-- Open File -->

The Topics and the Entities

All we need to do now is define an XML application file with a data reader for topic ComplexDataTopic and a writer to SimpleDataTopic:

<!-- Domain Library -->
    <domain_library name="MyDomainLibrary">
        <domain name="MyDomain" domain_id="0">
            <register_type name="Measurement" kind="dynamicData" 
                         type_ref="Measurement"/>
            <register_type name="SimpleMeasurement" kind="dynamicData" 
                           type_ref="SimpleMeasurement"/>
            <topic name="ComplexDataTopic"  
                          register_type_ref="Measurement"/>
            <topic name="SimpleDataTopic"   
                          register_type_ref="SimpleMeasurement"/>
        </domain>
    </domain_library>
<domain_participant name="Transform" 
                     domain_ref="MyDomainLibrary::MyDomain">
            <participant_qos base_name="QosLibrary::DefaultProfile"/>
                <publisher name="MyPublisher">
                    <data_writer name="MySimpleWriter" 
                                       topic_ref="SimpleDataTopic" />
                </publisher>
                <subscriber name="MySubscriber">
                    <data_reader name="MyComplexReader" 
                                       topic_ref="ComplexDataTopic" />
                </subscriber>
        </domain_participant>

<!-- Complete File -->

The Actual Logic

Once you have defined what your types are and how your entities are called, you just have to write a simple chunk of Lua code that will be executed by RTI Prototyper. Let’s have a look:

local myComplexReader = 
    CONTAINER.READER['MySubscriber::MyComplexReader']
local mySimpleWriter  = 
    CONTAINER.WRITER['MyPublisher::MySimpleWriter']
local instance        = mySimpleWriter.instance

myComplexReader:take()
for  i, sample in ipairs(myComplexReader.samples) do
    if (not myComplexReader.infos[i].valid_data) then
        print("\t invalid data!")
    else
        local deviceId = sample['deviceId']
        local avgTmp = 0
        local sum = 0;
        for i=1, sample['temperatures#'] do
            sum = sum + sample['temperatures[' .. i .. '].value'];
        end
        avgTmp = sum /  sample['temperatures#']

        --setting the instance
        instance['deviceId'] = deviceId
        instance['avgTmp']   = (avgTmp - 32) * (5/9);
        instance['unit'] = 1 -- C

        -- writing the simple instance
        print("Writing sample with avgTmp = " .. instance['avgTmp'] )
        mySimpleWriter:write()
    end
end

-- Open File

As you can see, you have to write very little code to transform your complex data type into a simple one using RTI Prototyper.

In the first 3 lines we are just getting the complex reader and the simple writer. We are then assigning the pre-allocated instance of the simple writer to the variable called instance.

Next we are doing a take and, if the received samples are valid, we iterate over them, getting only the field we care about and aggregating the data. (In this example, we calculate the average of all the temperatures sent in the original sample, and we ignore humidity and pressure.) We can also do some more intelligent transformations, such as transforming the incoming temperature from Fahrenheit to Celsius (line 20)!

Once we have what we need, we assign the values to writer instance (lines 19-21) and we write the sample (line 25 ). It’s that simple!!!

For your convenience I uploaded all the code and files to a GitHub repository here.

In the repo you will find the following directories:

  • Transformer: Contains both the XML and the Lua file described in this blog post. To run it just execute:
rtiddsprototyper -cfgFile Transformer.xml -luaFile Transformer.lua
  • EmitterEmu: Contains an XML file and a Lua file to emulate the behavior of your ISS module Emitter. To run it just execute:
rtiddsprototyper -cfgFile EmitterEmu.xml -luaFile EmitterEmu.lua
  • FancyReceiver: Contains an XML file and a Lua file to emulate your fancy dashboard running on board the ISS. To run it, just execute:
rtiddsprototyper -cfgFile FancyReceiver.xml -luaFile FancyReceiver.lua
  • LimitedReceiver: Contains an XML file and a Lua file to emulate your limited subscriber running back on earth. To run it, just execute:
rtiddsprototyper -cfgFile LimitedReceiver.xml -luaFile LimitedReceiver.lua

So, clone the repo, play with the examples, read the few lines of code, and you will right away understand the power of RTI Prototyper with Lua. If you want more information on RTI Prototyper check out the Getting Started Guide or the other blog posts about it here.

And if you have some time left check out the other solution for doing scripting with RTI Connext DDS using Python and node.js here.

Features and Enhancements and Support – Oh My! Reply

connext dds 5.1.0_wordCloud

As engineers we all know that product releases are no joke. The making of a great product is fueled by hard work, and lots of it. It takes teams of people working together to make something bigger and better than any one of them could do alone. The results generated by all of that hard work can be downright inspiring and impressive. But do you know what my favorite part of the release is? It’s the feeling when we ship it to the public. When all of the ducks are in order. Every person in the company, irrespective of their group, has contributed something necessary to getting the product to that point. Seeing it all come together? That’s my favorite part. I love it.

Did I mention that RTI Connext DDS 5.1.0 was released 2 weeks ago? It was!

To help ensure your transition to the new version is as seamless as possible, as well as keep you informed on the new features and enhancements you’ll see in version 5.1.0, I thought a post that served as a cheat-sheet of sorts might be helpful. We have put a lot into this release and I’m sure that you’ll love some of the great new features such as our new Built-in QoS Profiles and additional support for DDS-XTypes . For up-to-date product information and more information about the release, don’t forget to visit our Community Portal follow us on twitter .

I hope you enjoy 5.1.0. We certainly enjoyed all of the work we did bringing it to you.

Connext DDS 5.1 – Download. 

downloadButton

Community Newsletter – Special Edition. A Special Edition newsletter, focused solely on the 5.1 release, was sent out to all registered community members. You can find a copy of it here (although, you all should be registered community members so you would have already received and read this!).

Free Webinar + Slides/Replay Available.

photo (2)

David (left) and Jan (standing,right) during their webinar on February 27th.

David (our VP of PM) and Jan (VP of Engineering) presented a webinar where they dove into the new release and it’s capabilities – it was pretty awesome. If you missed out, don’t worry! You can view the replay (here), the slides (here), and if you have any questions about the webinar content – or the release! – you can post them on the Community Portal forum. We’d love to hear from you and help you out!

5.1 Blogs. A handful of posts were put together to inform you of some new features and capabilities in this release.

Additional content on the Community Portal.

Also be sure to check out the Connext DDS landing page on our main website. It’s been updated with all kinds of info on the release, including a video.