In some cases — like traversing wide-area networks — TCP support is really important. Network routers may not forward UDP packets at all, they may perform address translation, and/or they may require that “connections” be initiated first on one side of the network. That’s why RTI has been shipping TCP transport support for a while now, both as a native transport for application-to-application communication and with RTI Routing Service as a means to federate DDS systems (which internally may be using another transport such as UDP or shared memory). Today, that layering of RTPS on top of TCP is RTI-specific.

At the OMG technical meeting just concluded in Arlington, VA, RTI and others provided initial proposals for standardization of an interoperable TCP transport for DDS. RTI’s presentation in the OMG’s MARS group is available for OMG members. This will be an exciting extension of DDS interoperability — which now includes five implementations with proven records; look for another post on this — to the wide-area network. It’s still early — an RFP or RFC is yet to be issued — but expect to hear more in a couple of months.

• New C++ and Java Application Programming Interfaces (APIs) were recommended for adoption
• A DDS security Request for Proposals (RFP) was issued

New C++ and Java APIs

The new C++ and Java APIs are more user friendly and enhance source code portability between DDS implementations. They provide an alternative to the original DDS APIs, which were indirectly specified by mapping a UML model into the OMG Interface Definition Language (IDL) and then into a specific programming language (using standard IDL-to-language mappings).

The new APIs were created specifically for C++ and Java, not generated from IDL. Thus, they take better advantage of language-specific features and conventions. This makes them more intuitive and concise. The new APIs also improve portability by eliminating the indirection introduced by the IDL to language mappings. The new language bindings are concrete: there is no room for interpretation.

Existing DDS users do not need to worry about backward compatibility. RTI will support both the new and classic APIs. And, of course, applications using both APIs are fully interoperable thanks to the DDS wire interoperability protocol (DDS-RTPS).

DDS Security RFP

The DDS security RFP kicks off the process of defining standard Information Assurance (IA) extensions to DDS. This includes encryption, authentication, access control, labeling and tagging. Initial submissions are due in June 2011.

RTI is contributing a response to the RFP based on experience with our existing security capabilities and a prototype implementation. If you have input, please let us know.

Securty has been a major focus for RTI. We recently completed two security-related research contacts with the U.S. Office of the Secreteary of Defense (OSD), have a on-going contract with the U.S. Air Force, and are just beginning a new contract with the U.S. Navy.

DDS Interoperability Demo

Also at the OMG meeting this week, four vendors participated in a DDS interoperability demonstration. I’ll post more information on this in the next few days. [And here it is]

• Perhaps the most important milestone for DDS followers was the recommendation for adoption of the new specification Extensible and Dynamic Topic Types for DDS (sometimes abbreviated “DDS-XTypes”). I wrote about DDS-XTypes late last year; it will make it much easier for DDS users to upgrade and evolve their systems over time. It will also enable whole new categories of plug-and-play tools and integration components — such as database connectors, recorders, and bridging/routing services — to be developed in a way that’s portable and interoperable across DDS implementations, increasing the size and diversity of the DDS marketplace and increasing competition. RTI was a lead author of this specification, and a number of the capabilities defined in it are available in pre-standard form in RTI Data Distribution Service today.
• The meeting also saw the presentation of three initial submissions, representing six different companies, to the Web-Enabled DDS RFP that was released last year. (The previous link is only accessible to OMG members; sorry. When it’s finished, the specification will be publicly available.) This very strategic initiative will produce a specification for the integration of systems based on DDS with those based on web technologies. For the first time, it will be possible for SOAP- or REST-based client applications to consume data flowing along a DDS data plane, and for them to provide services back to applications in that plane, in a standards-based way with COTS technology. RTI is currently developing an implementation of this technology in parallel with the standard; this component, called RTI Web Integration Service, will be included in the Professional and Elite Editions of RTI Data Distribution Service. You can see a video demo of DDS-web integration, based on a pre-release version of this component, on YouTube.
• Work continues on further improving the portability and usability of DDS implementations with updated C++ and Java API bindings. The improved C++ APIs are already in the revised submission stage, and we expect them to be adopted within the next small number of OMG meetings. The improved Java APIs will be presented as initial submissions at the next OMG meeting, to be held in Minneapolis in June. RTI’s initial proposal will be based on the DataDistrib4J APIs, which we have made available in open source.
• Finally, the group had a lively discussion of DDS security, especially with respect to data tagging and labeling. We expect to take these discussions to a formal RFP in the coming months.

I think it’s a testament to the strength of the DDS ecosystem that while its core standards remain stable and support significant real-world deployments, at the same time the depth and breadth of that ecosystem continues to grow. It’s an exciting time to be involved with this technology.

One of the really powerful things about DDS is that it brings to distributed systems the same kind of type safety that you’ll find in local applications. In addition to reducing errors, this deep knowledge of data types can improve performance and resource usage by reducing the number of data copies in the system and easing integration with other field- and type-aware technologies, including relational databases and even Microsoft Excel.

But as systems evolve over time, type definitions can evolve too, and it’s important that applications that are already deployed don’t break as the types used by new applications change. It’s also desirable to ease the development of infrastructure or cross-cutting components — like tools, recorders, generic data routing and transformation facilities, and others — that shouldn’t be tied to specific data types. DDS users have been solving these problems in a variety of ways for some time, and some implementations address them already, but it’s time for a standardized solution.

To that end, the OMG is working on a new specification, Extensible and Dynamic Topic Types for DDS, that will provide additional capabilities for the following:

• A clarified and extended type system that incorporates keys and extensibility as first-class concepts
• An API for the definition of new data types at run-time without code generation
• A reflective API for the construction, inspection, and manipulation of data samples based on dynamic type definitions
• The ability to define data types declaratively using not only OMG IDL but XML and XML Schema (XSD) as well for easier integration with enterprise systems

The proposed specification will be discussed at the OMG Technical Meeting next month and some outstanding open issues addressed. I expect the proposal to be voted on and approved at a subsequent meeting not far in the future.

If your organization is an OMG member, you can access the in-progress specification proposal documents yourself.

So, while getting started supporting customers developing distributed edge applications with real-time needs at RTI, I found some expected, and some unexpected, differences in how developers built applications for the “edge” compared to the enterprise. While the (embedded) distributed application architects paid closer attention to the “physics” as devices were low powered, networks more complex and sometimes ad-hoc, and microseconds & memory mattered, the relative lack of attention that they paid to the information model was very interesting.

(Reference: Read a paper as a result of this insight: “How Does Your Real-Time Data Look?”)

The technologists in edge environments spend significant time tuning the network links, but they often miss significant opportunities to make optimal use of available bandwidth by not focusing (enough) on data modeling. I recently saw a “mission and performance critical” data model that is put on the wire with over six levels of nesting in its data structure…

This relative lack of attention to the data model, while regrettable, can be better understood if we take into account that until recently, edge devices were weak (could not collect or process enough information), few (not choking the network, though bandwidth is always an issue), or not (richly) context-aware (taking advantage of other information available on the network). Since only a (relatively) few bits were published on a “functionally light” middleware,  the information model was not very consequential.

However, as edge applications become more complex — from Command and Control to Monitoring — the science of tuning the information model for a distributed application can benefit from the advances in building information models for enterprise applications. With the devices and networks becoming more powerful, and with middleware such as RTI Data Distribution Service putting more intelligence on the network, the bottleneck more and more is not the hardware, or the capabilities of a high-performance or a functionally-rich middleware such as RTI Data Distribution Service, but the inefficiencies introduced by a poorly designed data model.

(Reference: Read this paper on what you can expect a modern high-performance middleware to do: “Is DDS for You?”)

What is required is that while designing the information model for the distributed application, architects of high-performance distributed applications sit with network middleware experts to ensure that the information model fully leverages the middleware’s capabilities, such as local processing (Content-Filtered Topics), message aggregation, time-based filtering, and using sparse types to only send the updated fields on the wire…