#TBT: From Predicting to Propelling the Industrial IoT Reply

If you missed it, you should check out the recent press release about RTI’s growth in the Industrial Internet of Things (IIoT). It’s really a great time to be RTI! Sure, from a business perspective all the vectors point the right way. But for me, the most exciting things in that press release aren’t numbers.  I’m more amazed that we get to play with so many futuristic applications. Carbots? Renewable energy? Smart healthcare? Hyperloop? Flying cars? Wind turbines? CT scanners? We got ‘em all. And new things show up all the time.

How can a small company like RTI play in so many areas? It’s Thursday, so in honor of #TBT, I thought I’d take a look back and see how we got here.

For those of you RTI history buffs, we sold our tools business to Wind River in 2005. It was 80% of our revenues at the time. So, in 2006, we had some money and great people, but not much in the way of products. So, we started looking for new trends. What did we find? Here’s an excerpt from our 2006 vision paper called “The Data-Centric Future”:

Truly profound technologies become part of everyday life. Motors, plastics, computers, and now networking have made this transition in the last 100 years. These technologies are embedded in billions of devices; they have melted into the assumed background of the modern world.

Another step is emerging in this progression: pervasive, real-time data. This differs from the Internet in that this pervasive information infrastructure will connect devices, not people. Just as the “web” connected people and fundamentally changed how we all interact; pervasive data will connect devices and change how they interact.

Today’s network makes it easy to connect nodes, but not easy to find and access the information resident in networks of connected nodes. This is changing; we will soon assume the ability to pool information from many distributed sources, and access it at rates meaningful to physical processes.  Many label this new capability the “network-centric” architecture. We prefer the term “data centric” because the change, fundamentally, is driven by a fast and easy availability of information, not the connectivity of the network itself. Whatever the name, it will drive the development of vast, distributed, information-critical applications.

If you change “pervasive information infrastructure” to “Industrial IoT,” that was a pretty good prediction. Too bad the name didn’t stick, but I suppose “PII” sounds more like the second act of a play or a stuttering math teacher than a technology revolution. Anyway, we thought it was coming soon, but 6 years later, we were still predicting. An RTI handbook entry from 2012 (still before the IoT and before the IIC launched the Industrial IoT in 2014) expanded on this a bit:

There is amazing value in distributed information. Connecting people to information transforms society – news feeds, weather satellites, and the Internet are only a few examples – timely information flow drives value in every industry and every endeavor.

However, current technologies only connect people at human speeds. There’s an entirely new opportunity to connect machines at physics speeds. Just as the Internet connected people and fundamentally changed how we all interact, a new “pervasive information infrastructure” will connect devices at speeds fast enough to drive distributed applications. RTI’s people and technology are the best in the world at delivering that data to the right place at the right time. We fundamentally connect complex systems at extreme speeds better than any organization on the planet.

Our technology enables tens, or hundreds, or thousands, or (soon) millions of processors to work together as a single application. Why does that matter? Because intimately-connected systems can do things that weakly-connected systems cannot. They can request and access data from far-flung reaches fast enough to react intelligently. They can read deeply-embedded sensors, use that data to control high-speed machines, and feed the results to the enterprise for monitoring and optimization. This powerful connectivity is a fundamental transformation that will make currently difficult things commonplace and currently impossible things possible. We are already working on many of these applications. Our work will help astronomers probe the deepest reaches of space, protect passengers from injury on tomorrow’s roads, make our nation more secure from attack, and improve the efficiency of renewable energy generation. RTI is leading the new wave of large connected systems, systems that work together as one.

So how can we play in so many futuristic areas? Call us lucky or call us prescient, but the IIoT has been our target for over a decade. RTI is influential in the IIoT because we got a head start. We’re no longer predicting. We’re now realizing the future.

So, what’s next? Our new tagline is “RTI lives at the intersection of functional artificial intelligence and pervasive networking.” Think about that one for a while. AI and connectivity are perhaps the most important trends for the next 40 years. They will combine to bring new wonders to light in every industry on the planet. The IIoT is much more than a name or a connectivity technology; it’s a new infrastructure for, well, everything. Connecting things together is powerful. Connecting those things to smarts is a whole new game. Carbots will improve transportation, autonomous hospital devices will take better care of patients and smart networks will make green energy practical. But, these are just the ENIACs of the IIoT. It won’t be long until every device on the planet is part of a connected, intelligent infrastructure. That infrastructure will make everything more efficient, more useful and friendlier.  These are exciting times indeed!

In any case, it’s an honor to be a leader in the “new” IIoT revolution. It’s even more of an honor to be associated with the technical visionaries at RTI that put us there.

My Internship Ride at RTI Reply

-Where talent meets innovation.

FRIDAY! It’s the first thing that comes to mind when I think about RTI. To almost everyone, Friday means the end of the week, the beginning of the weekend, late night parties and, of course, catching up on sleep. But for me, Friday and RTI have an entirely different connection, and I will explain why.

It starts with a cold, snowy Friday in February 2016. I was looking forward to the upcoming weekend – needed a break from the stressful set of assignments and projects for my courses at Penn – when I got an email from RTI saying that the HR department would like to schedule a call.

Immediately, my excitement started to build, since prior to this I had gone through a couple of interviews with RTI, including a coding round for the role of a Software Engineer Intern. I hoped this call would give me a reason to go out and celebrate on Friday. Hope turned into reality when I got the call saying that I was being offered an Internship role at RTI for the summer. Though they proceeded to tell me about the compensation, benefits, etc., all those words only passed tangentially over me. The only information I was excited about was the part that I got the internship I had been hoping for. The call ended at 6 pm that Friday, and yes, as I said before: a reason to go out and celebrate!

After gladly accepting the offer, I communicated with Jan, the VP of Engineering (and the Head of Research) over emails to deepen my learning about RTI and to keep in touch before my start date. We discussed the project and skills to deploy. It was supportive of them to answer my queries and encourage discussions.

Fast forward three months, and it was the end of my 2nd semester. I flew to Sunnyvale, California and prepared myself for the first day at RTI. Starting on a sunny day on May 11th, an initial welcome by HR set the ball rolling for the internship that was to go on for the next 15 weeks.

Fig 1: The first view of my cubicle.

After the initial formalities of joining, I was soon greeted by the VP of Engineering, having already spoken over emails with him, I could now link a face to the name. Furthermore, I was introduced to my mentor for the internship project. After a quick tour of the RTI office, I was soon taken for a welcome lunch by my mentor to a delicious place where we discussed a little bit of the project as well as more about RTI.

The very next day was exciting since all of us were going for a toast at the nearby ToGo’s to celebrate a recent product release. How awesome is it that you get to attend a toast party the very next day after joining? I was sinking into the moment. I imagined what would be in store for me the next day since it was a Friday, and Fridays meant the start of a lovely weekend. RTI didn’t fail to impress me on my first Friday as well since I was told that the company provides lunches every Friday. Yummy delicious food from the best restaurants around the Bay Area. WOW!

Soon enough, there was RTI Corporate training scheduled at HQ, and I was auto-enrolled to join. The training christened appropriately as “QuickStart,” was true to its title in getting me on track with RTI products quickly! The training focused on the products we offer and what they do and how. In addition, the hands-on exercises were truly a bonus. Receiving a training completion certificate from the presenting Principal Solution Architect was truly icing on the cake. Time started to roll by, I started to work on my project, and planned and chalked out a schedule of how and when things were to be done.

My project was to develop an Eclipse plugin from scratch. The project entailed completely encompassing the steps that a user of RTI Connext® DDS would go through in generating code from an IDL, bringing the project to Eclipse, modifying the IDL and regeneration of support code and finally building and executing the code without leaving Eclipse or using any other tool/software.

It was very exciting for various reasons. Firstly, I was to develop this from scratch, which became a true learning experience since I could research the various ways to go about this and choose the best one that would suit our requirements. Secondly, the complete code base was to be done in Java, a language with which I’m comfortable and had a quest to learn in-depth. Developing this plugin resulted in a fantastic learning experience with Eclipse and Java. I also ensured that I followed industry standard coding practices and guidelines along with appropriate documentation and comments in the code base.

Carrying on further, I started developing prototypes and very proactively communicating all of the developments and progress on my project to my mentor and the VP of Engineering on a regular basis. This communication was well appreciated and helped them to track progress without any follow up from their end.

One thing I’m particularly fond of is RTI’s daily lunch time. All of us gather around and go to the food options located in the adjacent building; we buy ourselves food and get back to the office kitchen to eat. This culture undoubtedly creates one of the best times of the day since you get to sit and talk to everyone about various topics ranging from sports to technology to personal favorites (in my case, cars! 😊 ). I was pleasantly surprised and happy to understand that the regular corporate hierarchy and designation-based differences are non-existent at RTI and everyone is extremely friendly and amiable. How cool is it that our CEO plays ping pong with you and joins you at the lunch table for interesting conversations? Another fun thing to add here is the after-hours get-togethers. One such shown below:

Fig 2b: After hours at Eureka.

After successfully achieving the first milestone in my project schedule, additional milestones and tasks were added. Thinking from the perspective of the end-user, I started to analyze various other requirements and added them to the list of tasks for the next milestone, giving me a sense of accomplishment and satisfactory progression on a task to complete.

Soon enough, I showcased my work to our CTO, who patiently sat with me, went through every step of the project and appreciated me for things I did intuitively. He also provided suggestions that could improve the quality of the plugin being developed. Incorporating the suggestions as I progressed, I was beginning to feel a sense of fulfillment to the project towards the end of July.

Fig. 3: Figurative form of thoughts in my head 😝

Towards the end, when I had achieved something substantial, I was then given a slot at the weekly R&D meeting to show a demo of the Eclipse Plugin to my colleagues at HQ as well as to my colleagues in our Spain office via a video call. I received valuable input after the demo which helped me shape my project appropriately.

Two months had passed by and it was the end of July. One fine day, the VP of Engineering walked up to my desk and asked me to join him for a short walk around the building, wherein details of my plan to complete my courses for graduation and timeline were discussed.

A few days later, a meeting with HR provided a little more information which gave me a deeper hope about returning to RTI as a full-time employee. I was thrilled. I was waiting. At the end of July on a Friday morning, I was as usual busy with my semi-colons and code, when The VP of Engineering walked up to me with a paper in hand, and said, “Let’s discuss a few things.” I tried to keep my excitement from appearing on my face, as he revealed that RTI was making me an offer to return as a full-time employee and went over the details about the compensation and benefits.

That moment, just like the moment many, many Fridays ago in February, had a deja-vu effect, as everything spoken that day was only remotely and softly heard by me as I was too excited to receive an offer from RTI. I love RTI, love the Bay Area for numerous reasons, and being able to come back was something I always hoped for. I quickly gathered my thoughts together, thanked Jan and RTI, and continued my work until the end of the day.

There were about three weeks left, and I completed all the tasks and milestones by the first week of August, leaving me with two weeks to complete all the testing and documentation. This Eclipse plugin was to be deployed and tested on three different platforms mainly Windows, Linux, and Mac. Hence, I had all three machines on my desk, and my desk looked like this. 😝

Fig 4: My lovely desk with my toys.

On the final day of my internship, having completed the exit formalities, I was pleasantly surprised to receive a box full of my favorite brownie bites from one of the friends I had made at RTI. Another lunch from RTI, on my last Friday of the internship, completed the schedule. I was all set to fly out of the Bay Area that evening, and onward to India for a short vacation. Appreciating the help and support from many of my newfound colleagues, I bid goodbye to everyone at RTI and the Bay Area with one thought; I’ll be back!

Fig 5: Me with the RTI banner at the entrance

Three Simple Steps to Achieving Peak DDS Performance Reply

RTI Connext® DDS provides an order of magnitude performance improvement over most other messaging middleware. But occasionally we run into customers who are trying to improve the performance of their DDS communications. This performance improvement can be achieved in either throughput or latency. In this blog, I will go through the three simple steps required to assess the performance of your system and will also review some of the most common ways customers have improved performance of their DDS communications.

Step 1: What performance should you be getting?

Compare the numbers you are getting with the comprehensive DDS benchmarks that RTI provides here: https://www.rti.com/products/dds/benchmarks.

If you are not getting close to the numbers you see in the DDS benchmarks, there are a couple things to try:

Use RTI Perftest to make sure you’re comparing apples to apples.

The configuration of the NIC and the network switch, as well as the maximum network throughput and the CPU, all have an impact on the final DDS performance results. So, to make a fair comparison run the DDS benchmarks on your hardware.  RTI makes the DDS benchmark program, “RTI Perftest,” available in source code format with complete documentation.  You can find a copy of Perftest here:  https://community.rti.com/downloads/rti-connext-dds-performance-test

Make sure you are running your tests using the network interface you think you are using.

DDS enables shared memory and UDPv4 transports by default. If Shared memory is available between two nodes DDS will use that by default. But if there are many network interfaces available DDS will only use the first four. I’ve seen developers want to test out a certain network interface, say Infiniband, but it was not one of the first four listed and so DDS was not adding it to the mix. In fact, on Windows systems, the order that network interfaces are listed by the OS, and thus selected by DDS, is random and so the network interface you are actually using can change from run to run. In fact, DDS will actually send the same data over two paths, if they exist, to the same endpoint. This can take up CPU time and slow throughput.  You can explicitly select the interface you want (or do not want)  using the transport QOS “allow-interfaces” property.   Here is a good RTI Community article on the subject: https://community.rti.com/howto/control-or-restrict-network-interfaces-nics-used-discovery-and-data-distribution.

Following is the actual XML code for “allow_interfaces” and “deny_interfaces” QOS that lets you explicitly pick the network interface you want to use or do not want to use:


Step 2. Use the RTI DDS tools to diagnose your performance issues. 

Use RTI Monitor to look for the number of ACKs, NACKs, dropped packets, and duplicate packets.  If these numbers are high, it can be due to several things:

  • Transport buffer sizes are too small
  • MTU  is not optimized for switch
  • There may be too many heartbeats causing multiple resends for single NACKs, indicating the reader is not keeping up
  • The CPU and memory process(es) are bound.

Use RTI Monitor or Admin Console to compare QOS settings of the DataReaders and DataWriters.  Sometimes you are not using the QOS values you think you are using.

A great way to learn about using the Admin Console and the Monitor tools is to watch this video from our tools lead, Ken Brophy.

Step 3. Now let’s start to look at your application to see how we can speed things up by changing the “shape” of the data in motion.

RTI DDS gives you many ways to fine tune your system using QOS settings. This flexibility is great because you have a lot of control over how DDS works.  But all the options can be daunting! I won’t go over every setting (this blog would quickly grow to be a textbook) but I will hit on what I feel are the most important settings to check in regards to performance.

First, don’t use strict reliability if it is not needed. Strict reliability makes sure that every sample reaches every reliable destination and will re-send samples if necessary. Resending samples and the structure that supports them take time and memory.  Many applications would be fine missing a sample very occasionally or waiting longer for it to be re-transmitted.

If you do need to use strict reliability then start with the DDS built-in profile “StrictReliable.HighThroughput”.  It is a good idea in general to use the built-in profiles that RTI provides. These built-in profiles are set up by RTI to have all of the default settings needed for the most common DDS use cases. The built-in profiles can be used as-is or can be used as the basis for your QOS configuration and then tweaked for your specific needs. You can read about using DDS built-in profiles and get a working example here:  https://community.rti.com/examples/built-qos-profiles 

Using Extensible types (XTypes) and sequences of structures can hurt performance. DDS serializes and de-serializes data it sends and receives, and this process takes a lot longer with complicated data types.

Adjust heartbeat_period/ ACKNACK combo.  In reliable communications, the DataWritersends DDS data samples and heartbeats to reliable DataReaders. A DataReader responds to a heartbeat by sending an ACKNACK, which tells the DataWriter what the DataReader has received so far. In addition, the DataReader can request missing DDS samples (by sending an ACKNACK) and the DataWriter will respond by resending the missing DDS samples. So, the heartbeat_period can control how quickly a data reader can acknowledge receipt of a sample or ask for a sample to be re-sent, impacting performance.  Here is an article that talks about how the heartbeat_period can impact latency and throughput.

Modify the Asynchronous Publisher configuration to use flow control to lower the data rate. Sometimes if the data rate from the writer is too fast, the reader gets swamped and the resulting dropped samples and resends slow down the system. Lowering the writer’s data rate a little leaves room for repairs, etc. This gives DDS time to handle incoming data and avoids costly resends. You can use a flow controller to shape the output traffic your publisher will generate. By using an asynchronous publisher and custom flow controller you can lower the data rate. You can see a working example of how to use the asynchronous publisher here: https://community.rti.com/examples/asynchronous-publisher

For smaller sample sizes, use batching and/or Turbo Mode. Batching groups of small samples into a single large packet is more efficient to send and can result in a large throughput increase. But note that while the use of batching increases throughput, it can hurt latency when little data is being sent (because of the added time needed to batch small samples). In high-throughput cases, though, average latency results because of all the CPU saved on the subscriber side of the interface.

Turbo Mode is an experimental feature that uses an intelligent algorithm that adjusts the number of bytes in each batch at runtime according to current system conditions, such as write speed (or write frequency) and sample size. This intelligence gives Turbo Mode the ability to increase throughput at high message rates and avoid negatively impacting message latency at low message rates.

Here is an article that goes into detail on how to use batching and includes a working example: https://community.rti.com/examples/batching-and-turbo-mode

Use multicast for topics with more than a couple of subscribers. Multicast allows a publisher to send to multiple readers with a single write, greatly reduces network and publisher-side processor utilization.  Note that sometimes this feature is not available at the network level.  Here is a good article on  how to implement multicast: https://community.rti.com/best-practices/use-multicast-one-many-data

For reliable communications modify the Send Window size. When a reliable DataWriter writes a DDS sample, it keeps that sample in its queue until it has received acknowledgments from all of its subscribing DataReaders that the sample was received. The number of outstanding DDS samples allowed is referred to as the DataWriter’s “send window.” Once the number of outstanding DDS samples has reached the send window size, subsequent writes will block until an outstanding DDS sample is acknowledged. Anytime the writer blocks, it hurts performance. You can read about adjusting the Send Window in section of the DDS User’s Manual.

Modify the transport settings. Whether you are using UDPv4 or shared memory or a custom transport, having the right buffer sizes and message sizes configured is extremely important when trying to optimize performance. Following is XML code for modifying transport message size and buffers sizes for the UDPv4 transport:


Note that the sizes used here are suggestions for optimizing performance when using large samples. You can make these values smaller for smaller samples.

I hope this advice is helpful in getting the best performance out of your DDS Application. I’ve listed the tips I’ve found most helpful for improving DDS performance but there are other methods that can also be helpful depending on the circumstances. In order to get more information on improving throughput or latency (or really help with any other Connext DDS issue), I encourage you to check out the RTI Community portal. The RTI Community portal is an excellent source of information and support! And of course, always feel free to contact our great support department or your local Field Application Engineer for further help.