Data distribution is about observing a changing world. A system whose communication is based on this paradigm tends to become data-centric: it becomes more concerned with modeling the first-class concepts of its business domain and less concerned with managing second-class “who-told-whom-to-do-what” middleware concepts like queues and messages. Along the way, it enjoys the benefits of decreased coupling and improved reliability, scalability, and performance.

Data Distribution and Its Kin

Classically, messaging is an evolution of the remote method invocation (RMI) paradigm — an attempt to make that paradigm less coupled and more scalable by making it asynchronous. A message says “I tell you to do this.” When compared with RMI, “I” and “you” are more abstract, both in identity and multiplicity, and the request can be queued for processing at a later time or by another party without making the sender wait. These are improvements, but the interaction remains coupled, because the roles of “I” and “you” (often in the guises of “client” and “server” or the trendier “service consumer” and “service provider”), as well as the intention of what action should be performed, are still very much in play.

Eventing, like data distribution, is preoccupied with changes to the world. An event says “I changed in this way.” It reduces coupling by entirely removing both the recipient of that information and any notion of intention from you business logic and your mental model; who might receive an event, and what they might choose to do as a result, are not the business of the event source. But state management remains a problem, because in order to understand the change that occurred, all recipients must have an up-to-date understanding of the state of the world prior to the latest event — “the price went up by a dollar” doesn’t do me any good if I don’t know what the price was before. This temporal coupling means that every recipient must process every event in order, whether those events are interesting or not, just in case the interpretation of a subsequent interesting event should happen to require the state established by a previous otherwise-uninteresting one.

The resulting processing and state management are complex and expensive. As a mitigation, they are frequently factored out of the applications that need the data and into state-management “servers” that “clients” must query using a message-centric or even RMI-based approach — a huge regression in engineering practice! The system becomes complicated by the presence of multiple interacting communication paradigms, and the servers (which serve no business role) introduce performance and fault-tolerance choke points.

A data-centric architecture eliminate these problems by simplifying the interactions. A data sample says simply “the world is like this.” It thereby eliminates coupling not only in terms of source, recipients, and their intentions, but also in terms of time. There’s no longer any need for recipients to process or store information they don’t care about, because samples don’t implicitly encompass previous samples. Therefore it becomes perfectly reasonable for one observer to examine the state of the world every second, or every minute, or every hour — and for another to observe every single intermediate state, even if those states change from one to the other many times a second.

Modeling the World with DDS

A set of DDS entities, and the data they distribute and manage, define a view into this changing “world.”

• A “domain” defines the boundaries of the world, the set of information that a collaborating group of applications might find interesting. A “domain participant” defines the presence of some application in that world; it is the data-centric analogue to what is frequently known as a “connection” in the messaging middleware.
• A “type” is a structural description of some part of the world — for example, an Antelope is brown in color and has four legs and two horns; a Ferrari is red in color and has four wheels and two seats. A type has a formal definition, usually (though not always) in a declarative language like XSD or OMG IDL, and it implies a corresponding definition in the target programming language.
• A “quality-of-service” (QoS) definition defines the fidelity with which some party/parties is/are able to describe the world. For example, will the description contain every state the world passes through or only a subset? Will observers have access to new states of the world only, or will they be able to see previous states as well? If the latter, how far back will those previous states go?
• A “topic” defines some aspect or subset of the world consisting of similar objects. As such, it combines a type, which defines the structure of those objects, with a QoS definition, which defines how they can be observed to change.
• An “instance” defines a single object in the group defined by a topic. For example, a topic may be used to distribute the positions of airplanes as detected by a radar. Each plane would be an instance. All radar tracks have the same structure (type) and are updated in the same way (QoS). But they are also distinct from one another: it matters whether the plane at a given location happens to be American Airlines flight 123 or Delta flight 456.
• A “data writer” defines a source of information about a particular subset of the world (topic). As such, it may override the QoS of its topic — multiple parties may provide information about the same part of the world but with different degrees of fidelity.
• A “data reader” defines an observer of a particular subset of the world (topic). As such, it may also override the QoS of its topic. Furthermore, it may only be able and/or interested to observe certain states of the world. For example, it may only be interested in airplanes flying over a particular geographic area or in stocks trading at over $20/share.

By creating a data reader with a certain QoS definition, an application makes an affirmative statement that it wishes to observe a certain portion of the world under a certain set of circumstances. For example, it may state that it is interested in observing the most recent five states (samples) to the objects (instances) in its part of the world (topic), but it doesn’t need to process changes more frequently than once every second.

This statement is one of interest only; it in no way requires the observer to actually observe a certain set of samples in a certain way or within a certain period of time. On the one hand, the observer may choose to be notified asynchronously of every new sample and to respond to it immediately. On the other, it may “go away” to other business and return hours later; when it does, it will find the most recent five samples of each instance, occurring no more frequently than once every second, waiting for it. In the mean time, DDS will have taken care of all of the necessary data reception, filtering, and replacement in order to make that happen.

DDS’s ability to combine notification and lightweight caching — in effect, to maintain an application’s observed state of the world on its behalf — is something no other standards-based technology provides. Developers of data-centric systems reap the benefits: higher performance and scalability, greater tolerance to dynamic network conditions, and ultimately improved ROI and time-to-market.

The answers to these questions are becoming increasingly important as real-time systems scale up and are integrated into Systems of Systems. Without the right underlying architecture and technology, applications will fail to satisfy their performance objectives. For mission-critical systems, the consequences of this could be catastrophic. Integration costs could also explode if proprietary and stovepipe workarounds are required to compensate for limitations in the underlying integration infrastructure.

You can download the white paper here.

Check it out and let us know what you think. What does “Real-Time SOA” mean to you?

• Perhaps the most important milestone for DDS followers was the recommendation for adoption of the new specification Extensible and Dynamic Topic Types for DDS (sometimes abbreviated “DDS-XTypes”). I wrote about DDS-XTypes late last year; it will make it much easier for DDS users to upgrade and evolve their systems over time. It will also enable whole new categories of plug-and-play tools and integration components — such as database connectors, recorders, and bridging/routing services — to be developed in a way that’s portable and interoperable across DDS implementations, increasing the size and diversity of the DDS marketplace and increasing competition. RTI was a lead author of this specification, and a number of the capabilities defined in it are available in pre-standard form in RTI Data Distribution Service today.
• The meeting also saw the presentation of three initial submissions, representing six different companies, to the Web-Enabled DDS RFP that was released last year. (The previous link is only accessible to OMG members; sorry. When it’s finished, the specification will be publicly available.) This very strategic initiative will produce a specification for the integration of systems based on DDS with those based on web technologies. For the first time, it will be possible for SOAP- or REST-based client applications to consume data flowing along a DDS data plane, and for them to provide services back to applications in that plane, in a standards-based way with COTS technology. RTI is currently developing an implementation of this technology in parallel with the standard; this component, called RTI Web Integration Service, will be included in the Professional and Elite Editions of RTI Data Distribution Service. You can see a video demo of DDS-web integration, based on a pre-release version of this component, on YouTube.
• Work continues on further improving the portability and usability of DDS implementations with updated C++ and Java API bindings. The improved C++ APIs are already in the revised submission stage, and we expect them to be adopted within the next small number of OMG meetings. The improved Java APIs will be presented as initial submissions at the next OMG meeting, to be held in Minneapolis in June. RTI’s initial proposal will be based on the DataDistrib4J APIs, which we have made available in open source.
• Finally, the group had a lively discussion of DDS security, especially with respect to data tagging and labeling. We expect to take these discussions to a formal RFP in the coming months.

I think it’s a testament to the strength of the DDS ecosystem that while its core standards remain stable and support significant real-world deployments, at the same time the depth and breadth of that ecosystem continues to grow. It’s an exciting time to be involved with this technology.

Click here for more articles on DDS. Click here for whitepapers on DDS, including a link to the Embedded Market Forecasters study that finds DDS middleware from RTI improves ROI when compared with other alternatives.

With the maturity and broad adoption of DDS, we are seeing a couple of trends.

• DDS is being used in larger and more geographically disperse systems
• Customers are moving to second-generation DDS based systems
• Users are integrating multiple systems that already deploy DDS as their underlying integration bus

To support these efforts, RTI recently introduced RTI Routing Service. RTI Routing Service provides a flexible solution for scaling DDS systems and for integrating disparate DDS applications. This includes:

• Applications that cannot directly communicate because they run on different networks (LAN and WAN), use different transport protocols (e.g., shared memory, IPv4 and IPv6), or are members of different security domains
• Applications that natively use different DDS data types, such as new and legacy applications, individual systems within a System of Systems, and applications that support different Communities of Interest (COI)

To learn how RTI Routing Service significantly reduces the costs of real-time system integration, upgrades and of implementing Cross-Domain Solutions (CDS), visit RTI’s web site or watch this video demonstration.

One of the really powerful things about DDS is that it brings to distributed systems the same kind of type safety that you’ll find in local applications. In addition to reducing errors, this deep knowledge of data types can improve performance and resource usage by reducing the number of data copies in the system and easing integration with other field- and type-aware technologies, including relational databases and even Microsoft Excel.

But as systems evolve over time, type definitions can evolve too, and it’s important that applications that are already deployed don’t break as the types used by new applications change. It’s also desirable to ease the development of infrastructure or cross-cutting components — like tools, recorders, generic data routing and transformation facilities, and others — that shouldn’t be tied to specific data types. DDS users have been solving these problems in a variety of ways for some time, and some implementations address them already, but it’s time for a standardized solution.

To that end, the OMG is working on a new specification, Extensible and Dynamic Topic Types for DDS, that will provide additional capabilities for the following:

• A clarified and extended type system that incorporates keys and extensibility as first-class concepts
• An API for the definition of new data types at run-time without code generation
• A reflective API for the construction, inspection, and manipulation of data samples based on dynamic type definitions
• The ability to define data types declaratively using not only OMG IDL but XML and XML Schema (XSD) as well for easier integration with enterprise systems

The proposed specification will be discussed at the OMG Technical Meeting next month and some outstanding open issues addressed. I expect the proposal to be voted on and approved at a subsequent meeting not far in the future.

If your organization is an OMG member, you can access the in-progress specification proposal documents yourself.

Message-centric design and data-centric design are similar, but they also differ in important ways. Let’s start with some terminology. There’s a reason why DDS says “sample” where JMS says “message”: those words are intended to suggest a different mental model to you.

Data-Centric Thinking

When you hear the word “sample,” imagine some data value (perhaps the temperature in a certain place) being published periodically. Those values describe a single logical data object — an “instance” in DDS terms — whose state changes over time. Or think of subsequent frames of a movie: the movie is a single logical object, and you view slices of it one after another. In other words, in a data-centric design, as the system’s state changes, it uses the middleware to publish that state directly.

Therefore, a data distribution middleware — such as RTI Data Distribution Service, which implements the DDS specification — needs to understand what that state is and under what circumstances it should be published. That’s why RTI Data Distribution Service allows you to describe your application’s own data types to the middleware (using XML, IDL, or a programmatic API): your application stores its state using those types, so the middleware needs to understand them in order to publish that state. That’s also why DDS-compliant middleware provides applications with such extensive control over which data should be published where and when, including filters on data contents and update rates, duration- and depth-based time-to-live rules, data reliability and availability contracts, deadline enforcement, and so on. A data-centric system uses its network middleware as a lightweight cache to manage its state, thereby replacing programmatic application logic with declarative descriptions of its delivery obligations and expectations.

Message-Centric Thinking

In contrast, a messaging middleware provides no facilities for state maintenance or management. Instead, the system maintains that state externally to the middleware, and when it changes, it sends “messages” about those state changes. The recipient(s) of those messages then decide if and how to update their own state. Because only the application-level logic “above” the middleware has access to its state and knows when and how to update it, there’s no need for the middleware to understand the contents of messages. Messaging middleware implementations therefore typically don’t support content-aware message handling and provide more limited control over delivery contracts than do data distribution middleware implementations.

Choices, Choices

Of course, the lines between these communication paradigms are often blurred, and many complex systems will use both in different places. For example, you may use RTI Data Distribution Service to distribute your data, but use a relational database to maintain and manage that data, and then use a product like RTI Real-Time Connect to make sure changes that occur on one side are automatically propagated to the other. Or you may implement a data-centric system with RTI Data Distribution Service but view that data, in certain subsystems, through a message-centric API with RTI Message Service.

Just as you can write procedural code in C++ or Java by choosing not to use certain object-oriented features of those languages, you can implement a message-centric system using a data distribution middleware by simply choosing to limit the capabilities of that middleware that you leverage. And just as you can write object-oriented code in C if you put your mind to it, you can implement a data-centric system using a messaging middleware. But to get the most value from your middleware, and save yourself the most effort, consider which paradigm will serve you best most often, and pick the right tool for the job.

One of the most interesting — and as far as I know, unique — capabilities of DDS middleware is that you can associate type definitions with your data streams. (For an introduction to DDS concepts, see the RTI whitepaper “Is DDS for You?“) Most other messaging systems do one of the following instead:

1. Use opaque data only, and make the application handle marshalling/serialization itself, including data encapsulation, endianness conversions, and the like.
2. Include complete data structure information with every message, including the names and types of any fields it contains.

Multi-vendor interoperability is fiction without some kind of type system, which disqualifies (1). How can I so much as send you a single string of text if we can’t agree whether that string should be length-prefixed or NUL-terminated, how wide the characters should be, or what encoding they should use? But self-describing messages are awfully heavy on the network, and they fail to capture generalities that already exist in your system.

The fact is, whichever component inserts a data value into a message almost certainly knows the name and type of that data value, and whichever component receives the message has some expectation about the names and types of the values it will find inside. What DDS gives you is the ability to expose that meta-information so that you can derive some value from it.

• The programming language you use to interface with your middleware probably supports strong typing. With DDS, you can extend that type system — including the definitions of your application’s specific data types — across the network.
• The database to which you connect has a type definition implicit in its tables’ schemas. Wouldn’t it be nice if the same information model made sense when your data was in-flight as when it was recorded?
• Use a CEP engine? Leading implementations rely on SQL-like languages to inspect and correlate data values based on well-known fields within the data.
• Some people like to pull their data into Microsoft Excel to analyze and visualize it. There are those rows and columns again: I’m noticing a pattern.

When you declare your data type up front, the middleware can also share that type’s definition up front — instead of with every message — so that you enjoy all of the benefits of type awareness as well as those of data compactness. Who says you can’t have your cake and eat it too?

But that’s not the subject of this post.

What you may not have noticed is that RTI Message Service is also the easiest JMS implementation to set up and administer. In fact, it requires almost no administration at all.

There are no servers or daemons you need to manage, and there are no servers or daemons that can crash and bring down your system. Just put our libraries on your classpath, set up your JNDI repository, and you’re ready to go.

That JNDI repository, by the way, is just an XML file. Edit it with any tool you want, put it wherever you want. Just point your “provider URL” to it and you’re good to go. You don’t need a separate naming service running anywhere.

It’s even easy to evaluate. Click the link, give it a shot, and let me know what you think.