As American as Tapas and Apple Pie Reply

Every morning, double decker bus after double decker bus shuttles engineers from all over the Bay Area to the GooglePlex, the Facebook compound, the Apple spaceship or the Yahoo campus. Yahoo infamously ended its work from home privilege. Google pulls out all stops to bring engineers together in the same and crowded place, showers them with perks, all to make magic happen.

“It is best to work in small teams, keep them crowded, foster serendipitous connections” – from How Google Works

What do you do when you are not one of these tech behemoths? What do you do when the skillset you seek is highly specialized and all over the world? When you find the masters in building real-time distributed systems, do you require them to move to the Bay Area, one of the most expensive places on earth? Housing prices throughout the larger Bay Area are astronomical. Do you make them slug through the busy Bay Area commute, even if that means that some will sit in traffic for over an hour each way?

As a small software company building real-time infrastructure software (“data distribution software”) for the industrial internet of things (IIoT), headquartered in Sunnyvale, California, we opted for a hybrid approach. We have two main development sites: Sunnyvale, CA and Granada, Spain.  Engineers have a flexible schedule to arrive at work to avoid the busy commute times. They all have the option to work from home occasionally. Most often, they choose a fixed schedule to work from home: e.g., every Wednesday. We also have remote engineers all over the US: Massachusetts, Florida, New Hampshire, New York, Virginia, and Minnesota. Managing the team, I stress about how to blend this team together, as if they were all in the same location and “foster serendipitous connections”? How do you cross the many time zones and yet leave no team behind?

The key to making this work is to establish the right team habits which build trust and are based upon transparency. We experiment often with new collaboration tools. If these tools do not foster more transparency or build trust, they will fail. Use tools and establish habits which emulate what happens in a traditional office setting, where you can drop into somebody’s office of a chat or for help. We’ve found that the following set of tools providing video, group chat and shared documents, paired with good team habits, to be the most effective.

When you want help debugging a problem, don’t default to sending an email. Email is impersonal and can get snippy (remember the Have you ever kissed a girl? email). Instead contact a remote engineer by video chat (You may still provide the log message via email). We use a lot of video these days, so much that we had to upgrade our internet to handle the many concurrent calls. We use Google Hangout for small group meetings, and installed a Google Chrome Box in several conference rooms. For example, our weekly bug-court meeting is a hangout with folks in Sunnyvale, New Hampshire, San Francisco, Maryland, Massachusetts and sometimes New York and Granada. For larger meetings, we use Webex and ask folks to turn on their camera when talking. In our large conference room, we have a remote controlled camera so remote participants can pan the room to the speaker. Yes, using video has it challenges (is my remote office clean?, CPU usage of Google Hangout, etc), but it is a key habit which help build trust.

An important practice are virtual (daily) scrums, through hangouts or group chat. We do this in individual development teams, and since recent, across the entire team. In individual development teams, the synch up meetings are more detailed and cover progress, plans, specific blockers or needs. When it comes to the entire team, we ask folks to post in a group chatroom specifics of what they will be working on that day. Initially we experimented with IRC (which failed on some platforms as it wasn’t as easy to use), and now we use Atlassian HipChat. Our rules for using HipChat are simple:

  • Rule #1: When you start your day, you say a virtual good morning and mention what you will work on in the GoodMorningGang room. This is similar to walking into the office and chatting with your colleague of what they will be doing that day. No good morning, no more soup for you: you don’t get to be part of HipChat. This rule has brought folks a lot closer. You get a sense of folks are working on, what they level of stress and frustration is, and you get to celebrate and chit-chat, as if folks were all in the same office.
  • Rule #2: move the conversation to the right room: e.g., don’t launch a discussion about platforms in the GoodMorningGang room; take the platform related discussions to the All Things Platforms room.
  • Rule #3: memes and animated gifs are allowed and encouraged. All work and no play makes Jack a dull boy. It’s ok to goof off, have some fun and create silly memes or celebrate with a little dance.

Very little of what goes on in the engineering team is a secret in the team or in the company. All our weekly meeting notes, team summaries or discussions are posted in internally available Google Docs. The engineering meeting notes are posted weekly to the entire company. We use Atlassian’s Jira to keep track of what we work on, or what type of bugs people have encountered with our products. This is accessible to the entire company. During weekly tech briefings, we educate each other about cool technical developments in the development and research team.

There are many more habits and tools which helps us work more efficiently as a distributed team: from a reasonably flat organization where the people doing the work are encouraged to make the decisions, to transitioning to a better revision control tool (git), to being able to power cycle all embedded boards in our lab from anywhere in the world.

Making a remote team work efficiently as if they are in the same office is not easy and takes constant adjustments and experimentation with habits and tools. When experimenting with a new habit or tool, always start small. Create pockets of excellence, succeed and then copy to another group. Be patient in the process. The combination of restlessness (aka we need to look for better tools to work as a distributed team) and patience (make them work) is important. Live the behavior.

RTI in Condition-Based Maintenance Showcased at NIDays Reply

NIDays: CBM university south carolina and RTI

It’s been a while since the last NI event in Austin, NI Week 2014, where RTI made a splash with its easiest to use and learn DDS offering, RTI DDS Toolkit for LabVIEW. RTI demonstrated its Python DDS bindings working with RTI DDS Toolkit for LabVIEW and Lego Mindstorm NXT robots simulating a closed loop control system. During NI Week 2014, we heard and engaged in many of the discussions on condition-based maintenance (CBM), especially in energy vertical markets. CBM has become such a hot topic, promising huge amounts of cost savings both for the industry and government. Upon returning from Texas one thing we knew was that we wanted to explore it further.

On November 18th, RTI and the University of South Carolina collaborated during NIDays 2014 in Washington, D.C. We demonstrated RTI’s Connext DDS in action as a condition-based maintenance (CBM) platform.

This time in Washington DC, instead of using our robots, University of South Carolina’s IGB miniature test stand was in action. We all had to google “IGB” to figure out what it stands for, “Intermediate Gear Box”. The test stand was equipped with the ability to run at 0 to 200 rpms with thermocouples and accelerometers, as well as condition indicators and sensors. It is approximately 50” in length, 16” wide, 16” tall, and weighs almost 150 lbs. For safety purposes, a Plexiglas enclosure was built to house the test stand.

During the demonstration, the sensor values on the IGB test bed were transferred to a PC via an NI Data Acquisition Device (DAQ). Then on the PC, RTI’s DDS Toolkit for LabVIEW published the sensor values to the RTI Data Bus. Data analytics and statistics tools, including IBM SPSS, received the sensor values via an RTI DDS C++ subscriber application and analyzed the data. By continuously monitoring the actual conditions, we demonstrated how predictive maintenance beats time-based preventive and failure-based corrective maintenance management methods.

All the folks who visited our booth walked away with handouts describing the demo and lots of food for thought about what they saw. This was a very successful step in creating awareness of what RTI can do to bring real-time to condition-based maintenance. Special thanks to graduate students at the Condition-Based Maintenance Center at University of South Carolina. Stay tuned for a video of students running the demo on the RTI and NI websites.

For additional information on our collaboration with the University of South Carolina on the CBM efforts, view the Press Release here or post your questions in the comments and we’ll be sure to get back to you!

Web Enabled DDS, The IoT, and The Cloud Reply


Web Enabled DDS, The IoT, and The Cloud all made an appearance this Halloween at RTI HQ. Notice that the IoT has a net with things connected and RTI is underlying the whole net of things. Very clever, Stan…

I need to find a better picture of our cloud, Brea. If you could see the signs that she’s holding, it really puts the entire thing over the top. Not that it needed much because that headpiece is amazing!


And if you follow us on twitter, you may have seen this already: Web Enabled DDS, aka Fernando Garcia. Get it? Get it?!

Happy Halloween, everyone!

Understanding RTI Connext DDS Secure Reply

Our Connext DDS Secure product is generating unprecedented interest. We rarely see so much demand for, and curiosity about, a product. It’s especially unusual because the product is still in Beta yet customers are nonetheless planning to ship it asap.  I thought I’d answer a few of the most common questions.

First, the new DDS Security standard specifies a security architecture and model. The Beta standard was adopted in March by OMG. We (RTI) chair the finalization committee; it should be final next year. RTI is first with support for the new standard. I’m sure other DDS vendors will also implement it, but nobody else has a product yet.

DDS Security is unique in the middleware space for several reasons. First, it addresses security more completely than other standards. The specification covers authentication, access control, confidentiality, integrity, non-repudiation, and logging. Second, it has a “plug in” design. The spec defines a set of standard plug-in components and an interoperable wire spec. But, you can define your own algorithms for the plugins. Finally, it protects DDS “topics,” not nodes or connections. So, it offers fine-grain control and can adapt to the unique Industrial Internet of Things (IIoT) requirements. It’s the first security standard that targets IIoT device-to-device and device-to-cloud networks rather than human or server-centric architectures.

Perhaps an example will make this more clear. Consider this (very) simple system:


Here, “PMU” represents a sensor (a phase measurement unit, common in power control).  The “CBM” (condition-based maintenance) analysis component is monitoring the system and looking for system health issues.  The simple operation of this system: the PMU sensor writes the state, the control reads that state and writes a set point. The CBM reads the state and writes alarms.  The operator can monitor the system.

In DDS, this system is easily set up as data flow between topics.  Of course, DDS specifies data rates, reliability requirements, and more.

To secure this system with Connext DDS Secure, you would create a configuration file that conveyed this:

PMU: State(w)
CBM: State(r); Alarms(w)
Control: State(r), SetPoint(w)
Operator: *(r), Setpoint(w)

This says, simply, that PMU can only write State.  Control can only read State and write SetPoint.  CBM can only read State and set Alarms.  And the operator can read anything and write the SetPoint (perhaps to turn off the system).  Connext DDS Secure directly enforces these very logical system constraints.

It really is that conceptually simple.  Of course, you still have to distribute certificates and the configuration file.  But, this “topic based” security is much more intuitive for IIoT systems than designs based on locking out protocols, or isolating nodes, or restricting access based on user roles.  Connext DDS Secure acts on the dataflow itself, directly and simply.

Importantly, our Connext DDS Secure product also doesn’t require any application code changes. You configure it & go. Connext DDS Secure offers practical, intuitive protection for existing systems.

Of course, no security protection is foolproof.  So, most all practical security systems combine protection (stopping bad things) with detection (finding and isolating breaches).  This is the reason, for instance, that your laptop has both a firewall (protection) and a virus scanner (detection).  Together, protection and detection provide much more secure systems.

DDS, being a software “DataBus”, also allows easy monitoring. We used that with PNNL to implement a “retrofit” security test for the power grid, replacing an old DNP3 line with a secure DDS line, thus implementing protection.  By tapping into the DataBus traffic and meta-traffic flow, we could then add a scripting capability (we have a slick Lua component).  Simple scripts could then detect many potential attacks, including compromised systems, man-in-the-middle attacks, etc. See

So, DDS lets you combine protection (the standard) with detection (through the DataBus).  Both are relatively simple to implement.

Our product is currently in early access release.  However, it is already undergoing fire testing.  Here is one very extensive test activity:

The USS SECURE cybersecurity test bed is a collaboration between the National Security Agency, Department of Defense Information Assurance Range Quantico, Combat Systems Direction Activity Dam Neck, NSWCDD, NSWC Carderock/Philadelphia, Office of Naval Research, Johns Hopkins University Applied Physics Lab, and Real Time Innovations Inc. USS SECURE’s test bed determines the best combination of cyberdefense technologies to secure a naval combatant without impacting real time deadline scheduled performance requirements.

As you can see, our security product expects some really demanding customers.  We can’t tell you much about these tests for obvious reasons.  However, I can say that I am very proud of our Connext DDS Secure product.  At this, and many other sites, it is proving extremely effective.

RTI Connext DDS Secure will be generally available next year.  If you have questions, please ask your local rep…

Connext DDS + Android Reply

Android is the dominant operating system in mobile, for both phones and tablets. As mobile eats away at the traditional desktop/laptop market, Android rivals Windows, Linux and Mac OSX as an operating system of major importance. Fueled by its application development eco system, Android is recruiting new developers at an unprecedented rate and those developers are innovating, creating novel application and taking Android into new places.

If we look past the Android Application Framework, we see an operating system that fits very well with the needs of embedded systems. Increasingly Android is being applied to such opportunities. So Android is becoming relevant, not only to the consumer aspect of the Internet of Things, but also to the industrial aspect. We expect to see a breakthrough in embedded device to cloud connectivity further driving new embedded applications.

To address the needs of developers, we have provided our distributed applications platform, RTI Connext DDS, on Android enabling the creation of publish and subscribe applications for the Industrial Internet of Things.

30dayFree trial rti connext dds professional

Scaling Down: DDS into Sensor Networks Reply

london connext conference rti EMEA 2014

Connext Conference 2014 is taking place 8-9 October in London. To learn more or RSVP, visit

Below is an abstract of a talk that RTI plans to have available at the RTI Connext Users Group meeting, to be held in London next week (

RTI is a firm believer in the Data Centric model of Pub/Sub. Even in places we can’t yet reach.

When it’s a question of scale, customers of RTI generally ask about the macro abilities of Connext DDS: how far up, or how far out RTI Connext DDS will scale, looking to support thousands, tens of thousands or even millions of discrete points-of-presence across their systems of systems. Sometimes this happens at design time, sometimes this happens after deployment.

On the other hand, sometimes it’s scaling downwards. RTI Connext Micro and Connext Cert support applications running on devices with limited size, weight and power requirements, but these systems still assume a certain level of hardware and capability.

Yet, there is still room for more — or in this case, less!

RheinMain University ( in Wiesbaden, Germany, has spent years looking at using the OMG DDS standard in the sub-RTI Micro domain. Here is Kai Beckmann, Dipl.-Inform.(FH), M.Sc., discussing what they’ve been working on.

sDDS – An adaptable DDS Solution for Wireless Sensor Networks.

Wireless Sensor Networks (WSN, cf. ZigBee, 6LoWPAN) are maturing into real world applications, from classical environment monitoring, home automation and the internet of things to industrial automation scenarios. There are improvements regarding available hardware resources, but the energy consumption is yet a limiting factor, and so are the heterogeneous software and hardware platforms. The Data Distribution Service (DDS) provides a standardised interoperable data-centric publish/subscribe architecture with real-time capabilities, suitable for many data-centric WSN scenarios.

However, DDS is rooted in larger-scale architectures – the full middleware functionality generally exceeds the available resources on an average sensor node – and the RTPS network protocol is not tailored to the small network frames common to WSN. We therefore propose sDDS, a DDS implementation for minimal embedded systems found in WSN. A Model-driven software development (MDSD) process is utilised to specify the system structure and applications requirements of the DDS functionality, and to generate individual sDDS implementations for each node. Furthermore, we present SNPS as an alternative transport protocol for DDS communication, particularly in WSN scenarios. SNPS has been designed as part of sDDS, optimizing for minimal footprint and compatibility between sDDS instances with different subsets of functionality.

Longer term, I would like to work with RheinMain University, with the purpose of bridging their SNPS transport protocol into an RTPS-based, full RTI Connext DDS system — giving the full scale system direct, data-centric access to the WSN for the purposes of aggregation, C2 and HMI, and giving the WSN direct, data-centric access to full scale systems for off-node processing.

If the idea of data-centric Sensor Networks interest you, there’s still time to register to attend.

DDS, Security, Smart People, Great Leadership, Fun Lunches, and James Gosling = Magical Internship Reply

ramya_headshotby Ramya Pradhan

I was at an NBA Orlando Magic game when I received news confirming the summer internship offer from RTI.  I knew from the first time I learnt about the company through my doctoral research on fault tolerance in distributed systems that RTI would be an awesome place to gain hands-on experience in my field of research.  The internship far exceeded my expectations in giving me ample opportunities to learn, contribute, and grow both professionally and personally.  The people, the work culture, and the profound impact RTI has in the fast emerging field of Internet of Things make it among the coolest places to be.  Come along as I reminisce over my experiences as an Intern.

I am a Computer Science doctoral student at the University of Central Florida.  I work on bio-inspired models for fault tolerance in distributed systems.  It was during one of my literature review sprints that I became aware of RTI through Dr. Douglas C. Schmidt’s research on fault tolerance.  The more I became acquainted with the research at RTI on distributed systems and its vast reach in the industry, the more I wanted to be associated with RTI.  I was overwhelmed when I saw internship openings for summer and lost no time in applying!  The interview process was quite rigorous, but I thoroughly enjoyed talking to the engineers who were more than passionate about their work.  I knew right then that RTI was where I wanted to be – at a place where your passion for what you do is nurtured aplenty!

On accepting the offer, I was delighted to know that I would be working on RTI DDS Connext Secure.  Secure systems  has always been one of my favorites.  I eagerly looked forward to experience the confluence of security and distributed systems.  I vividly remember my first day at work like it was yesterday. I was excited.  I was nervous.  I was a little uncertain.  I did have the theoretical know-how, but had little practical hands-on experience with distributed systems in the industry.  After completing the initial orientation meetings with the HR, IT, Accounting, and Safety departments, I was introduced to the Engineering team and to my mentor, Yusheng, over lunch.  It was a fine gathering in the kitchen.  It reminded me of family dinners at the table.  Little did I know then that this would be among the things I would miss the most!  Next was the meeting I was most looking forward to – a meeting with the VP of Engineering, Jan.  He introduced me to the workings of the RTI DDS Connext technology, tools, and potential projects that I would work on.  I was given all the material that I needed to ramp myself up to begin work as RTI’s Software Engineering Intern for Security.

I spent the next two weeks learning.  I watched presentations by the CEO, read orientation handouts, installed software with guidance from the build notes and a lot of help from the fantastic support team, and had whiteboard discussions with Yusheng to facilitate my understanding of the team’s expectations.  It was a completely immersive experience for me; part of it fueled by my curiosity, but mostly by my colleagues’ enthusiasm for the work they did.  Never once did I find anyone too busy to help me with obstacles along the way.  Thinking back upon this very aspect reminds me of an ancient Indian adage – “Yatha raja, thatha praja” meaning, “As is the king, so are his subjects”.  I strongly felt the CEO’s Servant Leadership style infused in all my colleagues; it was others’ needs first.

Smart People, Great Leadership: With CEO Dr. Stan Schneider, CTO Dr. Gerardo Pardo-Castellote, and the engineers.

Smart People, Great Leadership: With CEO Dr. Stan Schneider, CTO Dr. Gerardo Pardo-Castellote, and the engineers.

Upon completing the initial training, I eagerly set forth to explore my tasks.  I started each day with a list of things to work on.  I would settle down with a freshly brewed cup of coffee from the kitchen and start working off my list.  By noon, the Engineering team would all go to get lunch from the neighboring fast food places or the food trucks.  We would bring the food to the kitchen and sit at the tables – family style!  From then on, it was an hour of pure fun!  It was like unwinding with friends in the middle of the day filled with laughter, funny anecdotes, super heroes, history and traditions – Belgian, Italian, Spanish, Mexican, Indian, and Chinese, and work humor.  Oh, what fun it was!  It was truly rejuvenating to get back to work after lunch.  Each evening, upon my request, Yusheng and I had a meeting to discuss my work and progress. This ensured that we knew each other’s expectations and would have little gap in communication.  My day ended with most things taken care of from my to-do list.  In leaving for the day, I would eagerly look forward to the next day; that was the magic of the work culture at RTI.

The internship gave me a valuable exposure to the challenges that exist in the field of developing messaging systems for distributed systems, particularly in secure messaging.  The internship reinforced my belief that challenges are just fantastic opportunities waiting to be uncovered.  I am grateful to my mentor for letting me explore some of those fantastic opportunities.  Some solutions were easier to come up with than others, but most of all I enjoyed the process of coming up with solutions.  It typically involved a lot of whiteboard discussions, extensive analyses of code dependencies, thorough planning for integrating new code, and comprehensive debugging.  The most important takeaway for me from this process was that work could make one happy.

Adventures: Meeting James Gosling, visiting Chabot Space Center and Redwood National Forest.

Adventures: Meeting James Gosling, visiting Chabot Space Center and Redwood National Forest.

As I look back on the things that I have learnt and gained over the past summer, I am filled with gratitude for being given the opportunity to do so. Top of the list are motivating leadership, working with intelligent, dedicated, and passionate colleagues, challenges that seek creative solutions, the feeling that exudes ‘All for one and one for all’ at weekly engineering team meetings, the super fun lunch hour, FIFA world cup matches, and definitely, meeting James Gosling!  Thanks so much, Jan, for letting us know that James Gosling was in a restaurant near by!  Well, the Orlando Magic lost the game that day, but I won the most magical summer.  Thank you RTI for showing not only can one’s distributed system achieve impossible tasks when working as one but so can a company working as one – #1RTI.

ramya internship rti sunnyvale engineering

DDS Security: Completion of my first project under the able guidance of my mentor Yusheng Yang.

Leveraging W3C Org Documents (XML, etc) in RTI Connext DDS 1

london connext conference rti EMEA 2014

Connext Conference 2014 is taking place 8-9 October in London. To learn more or RSVP, visit

This is an abstract of a demo that RTI plans to have available at the RTI Connext Users Group meeting, to be held in London next month (

Working with structured data, each domain (Enterprise, Real-Time, Java ESB, etc.) will have its preferred method or defined structure for in-memory, and for stored data instances. The fun is when you move across the boundaries from one domain to another, or when you want to leverage one standard’s structures within a non-native domain.

The W3C consortium defines a Document Object Model to hold data in-memory, as well as XML and the Schema-for-Schemas, XSD, used when you want to serialize the data and communicate it to a third party. In OMG DDS, the equivalents are the Dynamic Data Object and Type-specific object PSMs, and RTPS serialization when transmitting instances.

Various ways to leverage the W3C standards in an RTI Connext DDS environment are demonstrated, including:

  • DCPS Discovery information to XML, useful for:
    • Self-documenting DDS Domains
    • Capturing snapshots of what is on a DDS Domain
  • Discovered Type information to XSD or XML, useful for:
    • Auto-generated Type info, formatted as XML for use with RTI Tools that can read XML-defined Types
    • Auto-generated XML Schema Documents, for XML files that can be used to pre-load instances with specific information (‘Configuration Propagation’, ‘Test’, and other use-cases)
  • Generating XSD information for use by Enterprise tools (that can’t read IDL)
  • etc

An example Java application is demonstrated, that uses DCPS Discovery Topics to generate XML and XSD files, as well as show how to read in XML files for use by DDS applications, either for remote configuration (when published as Last-Value-Cache for example), or for pre-loading Type instances for publication via:

ShapeTypeExtended steInstance =
"MyXmlInstances.xml", "YellowTriangleThree");

Service Provision and Discovery Reply

london connext conference rti EMEA 2014

Connext Conference 2014 is taking place 8-9 October in London. To learn more or RSVP, visit

This is an abstract of a demo that RTI plans to have available at the RTI Connext Users Group meeting, to be held in London next month (

The Oasis Reference Architecture for Service Oriented Architecture defines the concepts surrounding service ‘reachability’, including the use of Federated Service Registries (what services are known about) and Service Repositories (what you need to know, to access a service).  Once your Consumer knows what Services are available, and also how to attach to a given Service Interface, you’re now rocking that SOA.  So how does that apply to a real-time SOA, using DDS?

If you sort of squint at it, the DDS anonymous DCPS Discovery protocol is a part of the Service Registry.  For the Service Repository, Discovery also can supply you with the Types, but only supports the pub-sub integration pattern.  There isn’t any automated, or standards-driven, protocol for doing this using DDS.

To get a full, Federated Registry-Repository (cf Figure 38, page 63 of the reference document linked above), you can implement a Service Discovery process.  Using a well-known topic and type, you can search for suitable Service Interfaces (based on Search or Keywords, etc), determine what integration pattern a Service uses, find the Topics that are needed for it, and merge that information with the normal DCPS Discovery discovered information.  The end result is to build, ad-hoc, the necessary infrastructure to accesses that Service’s Interface — without requiring a maintained Repository.

For the London Connext Conference, RTI has a Raspberry Pi with an attached Pressure/Altimeter/Temperature sensor that publishes using pub/sub.  A service starts up and subscribes to the PAT data, which it then offers (as a Service Interface) via Request/Reply and “Last Value Cache” QoS.  The Service publishes a provision instance, loaded from an XML file, on a well known topic (“Service Provision”).  The instance includes particulars of the Service Interface:  This service uses Simple Request Reply, This is the Request Topic, the Reply Topic, with respective topic Type names, these are my human defined keywords.

A service consumer can now query the “Service Provision” topic using a content filter on the keywords, to find an arbitrary Temperature Service, identify that it is Simple Request/Reply, create the necessary Request writers and Reply readers (using the Types from normal Discovery process), make a request and process the reply.  The only foreknowledge the Consumer has, is that there may be a Temperature sensor on the network that uses a specific set of keywords.