Rapid Data Transformations Are Moments Away! 4

When I started college, everybody was talking about “Information Technology.” At that point I had been programming for quite a while and it wasn’t clear to me what coding had to do with that fancy terminology. After a few more years of coding, I realized the connection: all I do, day in and day out, is move bytes (information) from one memory location to another. Copying the contents of a struct into the socket buffer and sending it out; getting the bytes from the socket buffer and deserializing them into a structure to pass them to the application logic. Well, that’s part of what communication middleware does for you!

RTI Connext DDS implements the OMG DDS Standard. DDS is data-centric middleware. We call those bytes being transferred data, and we assign a type to them. The type describes each byte in your data, so you (and your application) can make good use of them.

Designing a good distributed system means having a good type system in place. Often though, those types carry a lot of information, and it can become pretty difficult to deal with them. Also, you may not need all the information all the time. This often happens when you have heterogeneous systems with nodes that have different capabilities, such as bandwidth constraints or resource limits. In this case, some of the nodes in the system may not be able to handle a whole sample for certain data types, but they still need to be able to receive part of the information.

Let me present a possible scenario first and then suggest one way to solve it.

scenario

Figure 1: Scenario

Let’s say a new testing module for the International Space Station (ISS) has been shipped to space. The module has a device that collects thousands of data points, puts them all in a DDS sample with a specific type, and writes it on the topic, ComplexDataTopic. We will call this device the Emitter.

One of the many different data points this device collects is the temperature from 10 different sensors. It puts all the collected temperature values into a sequence.

enum Unit {
   F,
   C,
   K
};

struct Temp {
    long sensorId;
    double value;
    Unit unit;
}

struct Measurements {
  long deviceId;
  // thousand of more fields. 
  sequence<Temp,10> temperatures;
};

// Open file

The module also comes with a FancyReceiver (what we call a topic subscriber) that runs a UI; it gets all the data contained in each sample and creates statistics and charts so astronauts can easily understand the data. For example, looking that the temperature data, it could calculate the average (mean) or the median, or spot significant differences between sensors.

Let’s now say that back on Earth, NASA scientists need to know what the temperature is on the testing module. They’re not interested in pressure and humidity. Just the temperature. But, since the bandwidth between Earth and the ISS “is what it is”, it’s considered acceptable to receive aggregates of the date. For example, instead of sending all the values for the temperature the testing module will only send the average, which will save the ISS some bandwidth on the link to ground. Basically, they want something that looks like this:

enum Unit {
   F,
   C,
   K
};

struct SimpleMeasurements {
    long deviceId;
    double avgTmp;
    Unit unit;
}

// Open File

We have many options to achieve this result:

  1. We can create an ad-hoc application: it will subscribe to the ComplexDataTopic, get the data we need, create another topic, another type, calculate what we need, and send it over.
  2.  We can use RTI Routing Service: write a custom transformation library and be done with it.
  3.  We can use RTI Connector via the RTI Prototyper with Lua.
  4. We can use RTI Connector via Python or node.js.

I will now explain option 3: Use RTI Prototyper with Lua to easily transform your types. If you are already an RTI Connext user, you will have what you need in your distribution (under the “bin” directory if you are using RTI Connext 5.2, or under the “scripts” directory for older versions). Otherwise, check out this page to learn how to get your free copy of RTI Prototyper.

We will create a new component, the Transformer. The Transformer will subscribe to ComplexDataTopic (with type Measurements) and will Publish to SimpleDataTopic (with type SimpleMeasurements).

The Types

For simplicity, let’s say that the complex type is the following:

<enum name="Unit">
  <enumerator name="F"/>
  <enumerator name="C"/>
  <enumerator name="K"/>
</enum> 
<struct name= "Temp">
  <member name="sensorId" id="0" type="long"/>
  <member name="value" id="1" type="double"/>
  <member name="unit" id="2" type="nonBasic"  
        nonBasicTypeName= "Unit"/>
</struct>
<struct name= "Measurements">
  <member name="deviceId" id="0" type="long"/>
  <member name="humidity" id="1" type="double"/>
  <member name="pressure" id="2" type="double"/>
  <member name="temperatures" sequenceMaxLength="10" id="3" 
        type="nonBasic"  nonBasicTypeName= "Temp"/>
</struct>

<!--Open File -->

The simple type looks like this:

<enum name="Unit">
  <enumerator name="F"/>
  <enumerator name="C"/>
  <enumerator name="K"/>
</enum> 
<struct name= "SimpleMeasurements">
  <member name="deviceId" id="0" type="long"/>
  <member name="avgTmp" id="1" type="double"/>
  <member name="unit" id="2" type="nonBasic"  
        nonBasicTypeName= "Unit"/>
</struct>

<!-- Open File -->

The Topics and the Entities

All we need to do now is define an XML application file with a data reader for topic ComplexDataTopic and a writer to SimpleDataTopic:

<!-- Domain Library -->
    <domain_library name="MyDomainLibrary">
        <domain name="MyDomain" domain_id="0">
            <register_type name="Measurement" kind="dynamicData" 
                         type_ref="Measurement"/>
            <register_type name="SimpleMeasurement" kind="dynamicData" 
                           type_ref="SimpleMeasurement"/>
            <topic name="ComplexDataTopic"  
                          register_type_ref="Measurement"/>
            <topic name="SimpleDataTopic"   
                          register_type_ref="SimpleMeasurement"/>
        </domain>
    </domain_library>
<domain_participant name="Transform" 
                     domain_ref="MyDomainLibrary::MyDomain">
            <participant_qos base_name="QosLibrary::DefaultProfile"/>
                <publisher name="MyPublisher">
                    <data_writer name="MySimpleWriter" 
                                       topic_ref="SimpleDataTopic" />
                </publisher>
                <subscriber name="MySubscriber">
                    <data_reader name="MyComplexReader" 
                                       topic_ref="ComplexDataTopic" />
                </subscriber>
        </domain_participant>

<!-- Complete File -->

The Actual Logic

Once you have defined what your types are and how your entities are called, you just have to write a simple chunk of Lua code that will be executed by RTI Prototyper. Let’s have a look:

local myComplexReader = 
    CONTAINER.READER['MySubscriber::MyComplexReader']
local mySimpleWriter  = 
    CONTAINER.WRITER['MyPublisher::MySimpleWriter']
local instance        = mySimpleWriter.instance

myComplexReader:take()
for  i, sample in ipairs(myComplexReader.samples) do
    if (not myComplexReader.infos[i].valid_data) then
        print("\t invalid data!")
    else
        local deviceId = sample['deviceId']
        local avgTmp = 0
        local sum = 0;
        for i=1, sample['temperatures#'] do
            sum = sum + sample['temperatures[' .. i .. '].value'];
        end
        avgTmp = sum /  sample['temperatures#']

        --setting the instance
        instance['deviceId'] = deviceId
        instance['avgTmp']   = (avgTmp - 32) * (5/9);
        instance['unit'] = 1 -- C

        -- writing the simple instance
        print("Writing sample with avgTmp = " .. instance['avgTmp'] )
        mySimpleWriter:write()
    end
end

-- Open File

As you can see, you have to write very little code to transform your complex data type into a simple one using RTI Prototyper.

In the first 3 lines we are just getting the complex reader and the simple writer. We are then assigning the pre-allocated instance of the simple writer to the variable called instance.

Next we are doing a take and, if the received samples are valid, we iterate over them, getting only the field we care about and aggregating the data. (In this example, we calculate the average of all the temperatures sent in the original sample, and we ignore humidity and pressure.) We can also do some more intelligent transformations, such as transforming the incoming temperature from Fahrenheit to Celsius (line 20)!

Once we have what we need, we assign the values to writer instance (lines 19-21) and we write the sample (line 25 ). It’s that simple!!!

For your convenience I uploaded all the code and files to a GitHub repository here.

In the repo you will find the following directories:

  • Transformer: Contains both the XML and the Lua file described in this blog post. To run it just execute:
rtiddsprototyper -cfgFile Transformer.xml -luaFile Transformer.lua
  • EmitterEmu: Contains an XML file and a Lua file to emulate the behavior of your ISS module Emitter. To run it just execute:
rtiddsprototyper -cfgFile EmitterEmu.xml -luaFile EmitterEmu.lua
  • FancyReceiver: Contains an XML file and a Lua file to emulate your fancy dashboard running on board the ISS. To run it, just execute:
rtiddsprototyper -cfgFile FancyReceiver.xml -luaFile FancyReceiver.lua
  • LimitedReceiver: Contains an XML file and a Lua file to emulate your limited subscriber running back on earth. To run it, just execute:
rtiddsprototyper -cfgFile LimitedReceiver.xml -luaFile LimitedReceiver.lua

So, clone the repo, play with the examples, read the few lines of code, and you will right away understand the power of RTI Prototyper with Lua. If you want more information on RTI Prototyper check out the Getting Started Guide or the other blog posts about it here.

And if you have some time left check out the other solution for doing scripting with RTI Connext DDS using Python and node.js here.

RTI Services Delivery Partner (SDP) Program Reply

Partnership

Core values are critical. They shape how and why we do what we do, both personally and professionally. Here at RTI, we do our best to make decisions that are informed by our values, which ensures our strategic goals are consistent with our mission – to enable and realize the potential of smart machines to serve mankind.

Our Services Delivery Partner (SDP) program is no exception. It’s the realization of one of our core values: valuing the importance of working as a team. As with any organization, the value of teamwork is realized internally as well as externally. With SDP, the value of teamwork is embodied in how we work with our partners, who are critical to continued growth. That’s one of the many reasons I’m excited about the recent addition of Tech Mahindra and Remedy IT to our rapidly expanding SDP program.

The Value of Partnerships

I find Wikipedia’s definition of partnerships quite helpful in light of our SDP program vision. Wikipedia defines a partnership as “an arrangement where parties agree to cooperate to advance their mutual interests…to increase the likelihood of each achieving their mission and to amplify their reach.” By leveraging the power of two brands and our market-leading connectivity platform for the IIoT, our SDPs can offer their customers a broader variety of product and service choices for a truly best-in-class IIoT solution. The SDP program gives partners the ability to design, plan, implement, and support RTI-enabled solutions in their customers’ environments.

Needless to say, we see these partnerships as critical to healthy growth, and that is precisely why we’re actively growing our SDP program and driving our strategic partnerships to the next level.

Advancing Our Mutual Interests, Amplifying Our Reach

Our SDP program strategy is specifically designed to focus on:

  • Establishing a growth multiplier
  • Amplifying our reach

Establishing a Growth Multiplier

Companies are continually looking for effective ways to strengthen their brand and expand awareness. We’re no different at RTI, and plan to leverage the power of our partnerships to increase our market presence and technology adoption rate. In fact, right after our joint press release was issued announcing our new relationship with Tech Mahindra, we saw immediate coverage:

Aligning with our SDPs will create a catalyst for growth that allows us to benefit from our partners’ solutions-focused global presence, significant vertical expertise, and worldwide sales teams. We see this as a perfect complement to RTI’s connectivity platform for the IIoT. These partnerships enable us to be even more aggressive in driving adoption of our technology in more and more vertical solutions. By combining efforts, we can deliver solutions and services for the emerging and dynamic requirements of the IIoT market in key verticals around the world, including energy, automotive, avionics, healthcare, and telecommunications.

Amplifying Our Reach

In addition to expanding the adoption of our connectivity platform, we’re also looking to amplify our services reach. Today, RTI’s professional services organization delivers high-value expertise, but is inelastic given our current composition. We plan to leverage the size and vertical expertise of our partners to offer our existing and emerging customers additional value, and establish a more elastic delivery capability, which is vital to RTI’s continued growth.

The Way Forward

As you can probably tell, we’re quite excited about our SDP program, and we look forward to providing additional updates as the program grows. If you have any questions or you’re interested in becoming a Services Delivery Partner, don’t hesitate to contact us at partners@rti.com.

TCP scalability improvements Reply

I’m excited to talk about new stuff we have added to our latest release, RTI Connext 5.2.0. In particular, I’ll talk about scalability improvements we added to the RTI TCP Transport Plugin.

Full Windows IOCP Support

RTI Connext can be used for a wide range of scenarios: from relatively small deployments with dozens of DomainParticipants communicating over a LAN, to WAN scenarios involving hundreds or even thousands of DomainParticipants. Our TCP transport provides a way to communicate over the WAN that is NAT-friendly and easy to configure. In WAN scenarios, the transport can be used in asymmetric mode. In this case, there is one DomainParticipant acting as a TCP server that receives connections from several other DomainParticipants acting as TCP clients. The server DomainParticipant does not need to know the addresses of the client DomainParticipants.

Something to consider when building a TCP server that handles hundreds or thousands of clients is socket management strategy, or socket monitoring strategy. This refers to how the system reacts to socket-generated I/O events.

While the strategy used by RTI Connext 5.0.0 TCP Transport Plugin (select() monitoring) had good scalability for servers running on Linux systems (up to 1,500 clients), we observed during experiments that scalability on Windows systems dropped to approximately 600 clients.

In order to solve this issue, RTI Connext 5.1.0 added partial support of MS Windows IOCP (I/O Completion Ports) socket monitoring strategy to the TCP Transport Plugin.

MS Windows IOCP provides an efficient threading model for processing multiple asynchronous I/O requests on a multiprocessor system. During our test on Windows systems running RTI TCP Transport Plugin, we observed an increase of the scalability from roughly 600 clients to 1500. However, RTI Connext 5.1.0 did not include Windows IOCP support for TLS over TCP.

I’m glad to announce that the latest release RTI Connext 5.2.0 includes full support of Windows IOCP socket monitoring strategy.

You can start using this feature by adding the following XML snippet to the TCP Transport Plugin configuration:

<participant_qos>
  <property>
    <value>
      <element>
        <name>{transport_prefix}.socket_monitoring_kind</name>
        <value>WINDOWS_IOCP</value>
      </element>
    </value>
  </property>
</participant_qos>
  • *Where {transport_prefix} is the name you assigned to the RTI TCP Transport Plugin (for example, dds.transport.TCPv4.tcp1).

External Load Balancer Support

Windows IOCP support is not the only new scalability improvement included in RTI Connext 5.2.0. Now I’ll talk about another addition that provides support for even more DomainParticipants acting as TCP clients.

When requiring high scalability using TCP, the TCP server becomes the bottleneck, and we need to take advantage of external load balancers. A load balancer is a software or hardware device that increases the capacity and/or reliability of a system by transparently distributing the workload among multiple servers. In particular, a TCP load balancer distributes connections from TCP clients among multiple TCP servers – in our scenario, DomainParticipants.

RTI Connext TCP Transport Plugin creates multiple connections when communicating two DDS DomainParticipants. These connections create a common state – a session – between the two DomainParticipants. In previous releases, the RTI TCP Transport Plugin was not friendly with externals load balancers. This was because a load balancer may have distributed the connections among multiple DomainParticipants , thus preventing communication.

diagram00

In order to support external load balancers, RTI Connext 5.2.0 includes a new session-id negotiation feature. When session-id negotiation is enabled (by using the negotiate_session_id property) the TCP Transport Plugin will perform an additional message exchange that allows load balancers to assign to the same DomainParticipant all the connections belonging to the same session:

diagram01

In order to enable the session-id negotiation, you can use the following XML snippet:

<participant_qos>
  <property>
    <value>
      <element>
        <name>{transport_prefix}.negotiate_session_id</name>
        <value>1</value>
      </element>
    </value>
  </property>
</participant_qos>
  • *Where {transport_prefix} is the name you assigned to the RTI TCP Transport Plugin (for example, dds.transport.TCPv4.tcp1).

This feature has been tested on a system using F5 Big-IP load balancer. However, it will work with other load balancers as long as:

  • They are able to modify the TCP data stream to include a unique identification of the node serving the first connection on a session.
  • They are able to assign connections to a server depending on the content of the TCP data stream.

I hope you find these new features useful and helpful in deploying highly scalable RTI Connext–based systems. To try it for yourself, download our free 30 day trial, today!