Standards-based DDS technology is a great way to implement this approach: it’s fast, dynamic, and fault-tolerant. The cost savings can be dramatic: one large distributed asset-tracking program replaced their proprietary middleware with DDS and realized a 90%+ savings while increasing capacity, performance, and robustness.

Check out the video.

Frans Ott says:

Good piece of work Rick! Does this mean that practising the DDS-perspective, I will respect the business domain model more as it is? And that, as a consequence, less domain-burden will be drawn into the middleware?

This question is an important and perceptive one. My answer has a couple of parts.

First, DDS is not a panacea. Adopting DDS but using it in the same way you used your old messaging middleware can bring you incremental benefits in terms of performance, scalability, and flexibility. But it cannot by itself provide you transformative benefits in terms of time-to-market and long-term maintainability.

Those larger benefits accrue as you adopt data-centric architecture — that is, as you begin to model your business domain and tie your distributed applications to that model instead of to each other. You can start to do this with any middleware, not just with DDS, and doing so will allow you to more smoothly integrate capabilities over time without perturbing existing behavior. But when you use a technology like DDS, which unlike other middlewares provides functionality to directly support this type of architecture, you get an additional benefit: the middleware can start to automate things that you used to have to do yourself.

There’s my second point: when you tell your infrastructure what you’re trying to do, it can help you do it better. I used the word “model,” which might sound frightening and esoteric. But everyone does this anyway! They just don’t call it that. When people design a system in a message-centric way, they write long documents (aka “models”) describing who has to send messages to whom, in what format, at what times. Then they email these documents to each other and implement the various components based on what the documents say. In the process, they write a lot of code to encode and decode data, validate it, monitor and respond to the system’s status, manage application state, and so on. And once they’ve done all of this work, they plug their components in and find out whether the documents had any ambiguities in them and whether everyone has correctly implemented all of those infrastructure pieces I enumerated.

The truth is, from the point of view of the real business logic, “the middleware” isn’t just a messaging or data-distribution component purchased off the shelf. It also includes all of the stuff your in-house infrastructure developers had to build on top of that to make the applications work right. A key innovation in DDS is the answer to this question: what if we took a lot of the ad hoc content out of all of those word-processing design documents and replaced it with a formal structure that the off-the-shelf middleware could understand? The answer is that the off-the-shelf component can start to do a lot of what the in-house “middleware-above-the-middleware” used to have to do — based on configuration rather than on custom code. The result is a more portable, interoperable system of higher quality and performance that was developed and integrated in less time. In other words: faster, cheaper, better.

You can use DDS as just a fast, interoperable way to send messages from here to there. Lots of people do. But DDS can do more than that if you’re willing to design differently. DDS does for data in motion something like what an RDBMS does for data at rest. It can’t make you create a good data design. But if you teach it what your design is, it can help you enforce that design, and it can reduce the amount of code you have to write in between your “middle”ware and your business logic.

Data distribution is about observing a changing world. A system whose communication is based on this paradigm tends to become data-centric: it becomes more concerned with modeling the first-class concepts of its business domain and less concerned with managing second-class “who-told-whom-to-do-what” middleware concepts like queues and messages. Along the way, it enjoys the benefits of decreased coupling and improved reliability, scalability, and performance.

Data Distribution and Its Kin

Classically, messaging is an evolution of the remote method invocation (RMI) paradigm — an attempt to make that paradigm less coupled and more scalable by making it asynchronous. A message says “I tell you to do this.” When compared with RMI, “I” and “you” are more abstract, both in identity and multiplicity, and the request can be queued for processing at a later time or by another party without making the sender wait. These are improvements, but the interaction remains coupled, because the roles of “I” and “you” (often in the guises of “client” and “server” or the trendier “service consumer” and “service provider”), as well as the intention of what action should be performed, are still very much in play.

Eventing, like data distribution, is preoccupied with changes to the world. An event says “I changed in this way.” It reduces coupling by entirely removing both the recipient of that information and any notion of intention from you business logic and your mental model; who might receive an event, and what they might choose to do as a result, are not the business of the event source. But state management remains a problem, because in order to understand the change that occurred, all recipients must have an up-to-date understanding of the state of the world prior to the latest event — “the price went up by a dollar” doesn’t do me any good if I don’t know what the price was before. This temporal coupling means that every recipient must process every event in order, whether those events are interesting or not, just in case the interpretation of a subsequent interesting event should happen to require the state established by a previous otherwise-uninteresting one.

The resulting processing and state management are complex and expensive. As a mitigation, they are frequently factored out of the applications that need the data and into state-management “servers” that “clients” must query using a message-centric or even RMI-based approach — a huge regression in engineering practice! The system becomes complicated by the presence of multiple interacting communication paradigms, and the servers (which serve no business role) introduce performance and fault-tolerance choke points.

A data-centric architecture eliminate these problems by simplifying the interactions. A data sample says simply “the world is like this.” It thereby eliminates coupling not only in terms of source, recipients, and their intentions, but also in terms of time. There’s no longer any need for recipients to process or store information they don’t care about, because samples don’t implicitly encompass previous samples. Therefore it becomes perfectly reasonable for one observer to examine the state of the world every second, or every minute, or every hour — and for another to observe every single intermediate state, even if those states change from one to the other many times a second.

Modeling the World with DDS

A set of DDS entities, and the data they distribute and manage, define a view into this changing “world.”

• A “domain” defines the boundaries of the world, the set of information that a collaborating group of applications might find interesting. A “domain participant” defines the presence of some application in that world; it is the data-centric analogue to what is frequently known as a “connection” in the messaging middleware.
• A “type” is a structural description of some part of the world — for example, an Antelope is brown in color and has four legs and two horns; a Ferrari is red in color and has four wheels and two seats. A type has a formal definition, usually (though not always) in a declarative language like XSD or OMG IDL, and it implies a corresponding definition in the target programming language.
• A “quality-of-service” (QoS) definition defines the fidelity with which some party/parties is/are able to describe the world. For example, will the description contain every state the world passes through or only a subset? Will observers have access to new states of the world only, or will they be able to see previous states as well? If the latter, how far back will those previous states go?
• A “topic” defines some aspect or subset of the world consisting of similar objects. As such, it combines a type, which defines the structure of those objects, with a QoS definition, which defines how they can be observed to change.
• An “instance” defines a single object in the group defined by a topic. For example, a topic may be used to distribute the positions of airplanes as detected by a radar. Each plane would be an instance. All radar tracks have the same structure (type) and are updated in the same way (QoS). But they are also distinct from one another: it matters whether the plane at a given location happens to be American Airlines flight 123 or Delta flight 456.
• A “data writer” defines a source of information about a particular subset of the world (topic). As such, it may override the QoS of its topic — multiple parties may provide information about the same part of the world but with different degrees of fidelity.
• A “data reader” defines an observer of a particular subset of the world (topic). As such, it may also override the QoS of its topic. Furthermore, it may only be able and/or interested to observe certain states of the world. For example, it may only be interested in airplanes flying over a particular geographic area or in stocks trading at over $20/share.

By creating a data reader with a certain QoS definition, an application makes an affirmative statement that it wishes to observe a certain portion of the world under a certain set of circumstances. For example, it may state that it is interested in observing the most recent five states (samples) to the objects (instances) in its part of the world (topic), but it doesn’t need to process changes more frequently than once every second.

This statement is one of interest only; it in no way requires the observer to actually observe a certain set of samples in a certain way or within a certain period of time. On the one hand, the observer may choose to be notified asynchronously of every new sample and to respond to it immediately. On the other, it may “go away” to other business and return hours later; when it does, it will find the most recent five samples of each instance, occurring no more frequently than once every second, waiting for it. In the mean time, DDS will have taken care of all of the necessary data reception, filtering, and replacement in order to make that happen.

DDS’s ability to combine notification and lightweight caching — in effect, to maintain an application’s observed state of the world on its behalf — is something no other standards-based technology provides. Developers of data-centric systems reap the benefits: higher performance and scalability, greater tolerance to dynamic network conditions, and ultimately improved ROI and time-to-market.

The answers to these questions are becoming increasingly important as real-time systems scale up and are integrated into Systems of Systems. Without the right underlying architecture and technology, applications will fail to satisfy their performance objectives. For mission-critical systems, the consequences of this could be catastrophic. Integration costs could also explode if proprietary and stovepipe workarounds are required to compensate for limitations in the underlying integration infrastructure.

You can download the white paper here.

Check it out and let us know what you think. What does “Real-Time SOA” mean to you?

Many developers suffer from confusion with respect to the differences between mutexes and semaphores. Michael Barr of Netrino provides solid information in his article Mutexes and Semaphores Demystified. My summary: mutexes protect shared resources by enforcing mutual exclusion; semaphores should be used for signaling across tasks.

Niall Cooling of Feabhas provides a longer discussion in two parts:

1. Mutex vs. Semaphores – Part 1: Semaphores
2. Mutex vs. Semaphores – Part 2: The Mutex

Cooling focuses especially on the problems that arise when developers use semaphores where they should rather use mutexes. In particular, he discusses accidental lock release, various causes of deadlock, and priority inversion. (The latter is of special interest to developers of real-time software.)

Although semaphores and mutexes are critical concurrency-control structures, the APIs to use them unfortunately differ from platform to platform with respect to both form and function. At the end of his Part 2, Cooling briefly summarizes some of the differences between the semaphore and mutex APIs of several operating systems. This section will be of interest to anyone developing systems that will be ported across very different platforms. Chris Simmonds goes into more depth, with respect to Linux in particular, in his article Mutex mutandis: understanding mutex types and attributes posted on The Inner Penguin.

Recording Data

The simplest case of persisting data in a distributed system is recording the data to a file system. This can be done locally for each node, or centrally on a “recording” node. Each system setup solves different problems. Persisting local state on each node allows you to recover from a local crash by reading that state during recovery. Recording all the data on a centralized node allows you to query the entire set of data recorded (not just the subset any node would see) to find patterns or anomalies, or do a post-mortem on a system failure.

Locally each RTI Data Distribution Service publisher or subscriber keeps samples in queues. To allow for local persistence a mirror of these in-memory queues can be kept on disk. After a crash the publishers and subscribers can then be created with the same state as they had before the crash.

Centrally you can run RTI Recorder that can be configured to subscribe to interesting data (or all data for that matter) and write it to a file system. This data can then be queried either in real-time or during off-peak hours. As an aside RTI Data Distribution Service and RTI Recorder have countless flight hours on mission critical systems!

Connecting the Distributed System to an Enterprise Database

In some cases simply having the data on a file system is not enough — the full power of an enterprise database is needed. At RTI we have solved this by having RTI Real-Time Connect for ODBC enabled databases. Now you can use RTI Data Distribution Service to implement a distributed database, or utilize a database for your persistent storage needs. This allows you to use any tool or query you are familiar with to work with a distributed set of data. The possibilities are literally endless!

No matter what the use-case — recovery, logging, post-mortem analysis or database integration — persisting real-time distributed data is here to stay. The challenge is persisting the data with minimal intrusiveness on the real-time aspects of the system. At RTI we work hard to make that happen.

Message-centric design and data-centric design are similar, but they also differ in important ways. Let’s start with some terminology. There’s a reason why DDS says “sample” where JMS says “message”: those words are intended to suggest a different mental model to you.

Data-Centric Thinking

When you hear the word “sample,” imagine some data value (perhaps the temperature in a certain place) being published periodically. Those values describe a single logical data object — an “instance” in DDS terms — whose state changes over time. Or think of subsequent frames of a movie: the movie is a single logical object, and you view slices of it one after another. In other words, in a data-centric design, as the system’s state changes, it uses the middleware to publish that state directly.

Therefore, a data distribution middleware — such as RTI Data Distribution Service, which implements the DDS specification — needs to understand what that state is and under what circumstances it should be published. That’s why RTI Data Distribution Service allows you to describe your application’s own data types to the middleware (using XML, IDL, or a programmatic API): your application stores its state using those types, so the middleware needs to understand them in order to publish that state. That’s also why DDS-compliant middleware provides applications with such extensive control over which data should be published where and when, including filters on data contents and update rates, duration- and depth-based time-to-live rules, data reliability and availability contracts, deadline enforcement, and so on. A data-centric system uses its network middleware as a lightweight cache to manage its state, thereby replacing programmatic application logic with declarative descriptions of its delivery obligations and expectations.

Message-Centric Thinking

In contrast, a messaging middleware provides no facilities for state maintenance or management. Instead, the system maintains that state externally to the middleware, and when it changes, it sends “messages” about those state changes. The recipient(s) of those messages then decide if and how to update their own state. Because only the application-level logic “above” the middleware has access to its state and knows when and how to update it, there’s no need for the middleware to understand the contents of messages. Messaging middleware implementations therefore typically don’t support content-aware message handling and provide more limited control over delivery contracts than do data distribution middleware implementations.

Choices, Choices

Of course, the lines between these communication paradigms are often blurred, and many complex systems will use both in different places. For example, you may use RTI Data Distribution Service to distribute your data, but use a relational database to maintain and manage that data, and then use a product like RTI Real-Time Connect to make sure changes that occur on one side are automatically propagated to the other. Or you may implement a data-centric system with RTI Data Distribution Service but view that data, in certain subsystems, through a message-centric API with RTI Message Service.

Just as you can write procedural code in C++ or Java by choosing not to use certain object-oriented features of those languages, you can implement a message-centric system using a data distribution middleware by simply choosing to limit the capabilities of that middleware that you leverage. And just as you can write object-oriented code in C if you put your mind to it, you can implement a data-centric system using a messaging middleware. But to get the most value from your middleware, and save yourself the most effort, consider which paradigm will serve you best most often, and pick the right tool for the job.

Let’s say you have a large distributed system where data is being exchanged between nodes and applications. You’re finding that some of your applications are getting starved for data, but everything looks fine in the network – there are only a few lost packets and plenty of bandwidth available. The next step is moving up the stack to see what’s going on with the networking middleware. Now, if you’re lucky enough to have a well-behaved middleware like RTI Data Distribution Service, there are ways to retrieve detailed statistics about the communication channels. Passing this information through a CEP engine you can write queries that determine when and where you have a slow producer or consumer in your system that is clogging everything up.

Another common use-case is event correlation. Seemingly disparate events may not be, and corroborating can trigger pattern based alerts. Anyone who’s watched a Hollywood spy movie knows this – increased cell-phone chatter coupled with certain keywords and the change in behavior of suspected operatives means something bad is likely going to happen. Using CEP events can be monitored and corroborated in real-time and alerts sent out when the situation warrants it.

CEP is an incredibly powerful technology, and with increasing data rates it is a technology that should be watched carefully in the next couple of years.

Go check out RTI Event Processing at http://www.rti.com/products/rtievent_processing/ for an integration between RTI Data Distribution Service and Complex Event Processing.

For infrastructure software, this “Out of box experience” can be a challenging task because a developer has to learn how to take this generic software that can do many things and apply it to solving his/her problems.  For software products that provide access through an Application Programming Interface (API), this usually occurs with a general program called “HelloWorld”.

In the realm of a distributed infrastructure this “HelloWorld” basically shows how to publish and receive a simple text message that contains “Hello World!”.  Using an application like this is very useful because it shows a developer what the basic minimum steps are to getting an application that can communicate.  And this does provide a little context on how to use their new API.  What this doesn’t do, however, is that it doesn’t show them how to get their application specific messages

Here at RTI, we take the concept of “Getting Started” to a new level with our Data Distribution Service (DDS) product.  With the use of a code generation tool called “rtiddsgen”, new developers can actually create applications that will publish and subscribe their own data types.  This utility, rtiddsgen, provides an option called: “-example <platform_architecture>”, and it enables the developer to specify a target architecture.  The target architecture can be Windows, Linux, Solaris, Java or a whole set of other Real Time Operating Systems.

Once the correct platform is selected the output from the utility is two applications with all the supporting code files needed to compile and run those applications.  The two applications are a Publication application and a Subscription application that are created specifically to the message type that is fed into the rtiddsgen utility.  Therefore instead of having just a simple “HelloWorld” application to start with, they now have an application that is created for any message type they can think of.  The resulting code is generated with lots of inline comments detailing the individual procedure steps, and also showing where in the code the developer can make changes to affect behavior.  If they want to increase the publication rate, it shows where to do that.  If they want to change or modify the data being published, it shows where to add that code.  If they want to act on the data instead of just printing it to the screen, it shows them where to add code.  Basically, it provides all the basic building blocks they will need to really make the applications part of their custom distributed framework.

With this “Easier” getting started capability, application developers can literally reduce hours and days off of their projected development time.  Therefore, the net result is that more time is spent on developing application specific “Business Logic” as opposed to infrastructure building blocks.

Among the support team, when we learn about a new debugging tool, we frequently joke about ‘our network debugging tool belt’. Over the years, this tool belt has grown to a nice variety of debugging tools, especially when you include platform specific debugging tools. We currently support a great variety of platforms: from the enterprise platforms (Linux, Solaris, Windows) to various embedded platforms, (including VxWorks, LynxOS and Integrity).

Below is a list of the tools we frequently use on support. I am leaving out platform specific tools, tools to monitor how a particular network stack is processing network traffic (e.g netstat) and general operating system debugging tools (vmstat, cpustat, lsof, gdb, strace etc.).

One of the most common questions on support is: why are my two DDS applications not communicating? The answer to this question falls into two categories: it is either a general networking / connectivity problem, or it is a configuration problem. Let’s take a look at the various tools we use to debug.

Tools to verify connectivity
1. ping verifies whether you have basic connectivity between two hosts. Not commonly known is that ping can also be used to generate multicast data. This is especially useful when your DDS discovery is using multicast and you are suspecting a multicast or time-to-live (ttl) problem.

For example, you can try to ping 224.0.0.1 which is the all-hosts group. If you ping that group, all multicast capable hosts on the network should answer, as every multicast capable host must join that group at start-up on all it’s multicast capable interfaces. There is one gotcha, by default when running a Linux kernel 2.6, the capability of ICMP to reply to broadcast/multicast ping message is disabled. Whereas, it is enabled by default in Linux kernel 2.4. You can enable it by disabling icmp_echo_ignore_broadcasts (reference: http://kerneltrap.org/node/16225)

If you want to experiment with the time-to-live value many ping implementations allow you to configure this via -t or -T.

2. traceroute (unix) or tracert (Windows) is a tool to visualize packet path used when sending a packet. It shows you the different hops / routers a packet traverses.

3. rtiddsping is a command line utility found in the scripts directory of your RTI Data Distribution Service installation. It basically does the same thing as ping, but not by using the ICMP protocol, but by using DDS. If you were able to ping the hosts and wonder whether the issue is a problem in how your DDS application is configured, then rtiddsping is a neutral application to verify whether the DDS ping /pong topic is working. Take a look at the various flags to modify the peerlist, ttl, multicast and reliability qos parameters.

Tools to analyze the data on the wire
4. One of my favorite network tools is probably Wireshark. Wireshark, previously called Ethereal, is a free packet sniffer and allows you to inspect the data on the wire. It will decode the packets based upon various protocol dissectors. The DDS wire protocol is commonly known as RTPS – real-time publish subscribe wire protocol. RTPS dissectors in earlier versions of Wireshark supported an early non-standard version of the protocol. RTPS2 is the dissector which supports the OMG standard of the wire protocol. RTI contributed the source of this dissector to the Wireshark community. We also offer it as part of our RTI Protocol Analyzer product. If you are using RTI Data Distribution Service 4.2 or later, use the RTPS2 dissector.

We use RTI Protocol Analyzer to debug discovery misconfiguration. You can observe which IP addresses are used during discovery announcements or which ports a particular node is contacting. You can also look at whether the reliability protocol needs to repair samples and how many. You are able to filter on Hearthbeats as well as corresponding ACK/NACK packets. Lastly the nice thing about it is that you can save it to a file to share with the RTI support team. We can partake in the analysis of why something isn’t working as expected.

Make sure you are monitoring the data both at the send and receive side. RTI Protocol Analyzer can only capture what is on its wire. If you can not install RTI Protocol Analyzer on the publisher or subscriber, hooking up a simple hub is a good option. A switch, unless you enable port mirroring, won’t do the job. (Doh, where did I leave my old hub now?)

5. snoop (Solaris) is a packet sniffer for Solaris. Unfortunately it does not dissect the RTPS protocol.

6. rtiddsspy is a command line utility, found in the scripts directory of your RTI Data Distribution Service installation, which allows you to automatically subscribe to publications in a specific DDS domain. You can observe both the discovery data as well as the user data. Take a look at some of the command line options to filter out particular topics.

7. RTI Recorder is a more complete data recorder. It can be configured to record specific topics and fields to a database which lateron can be queried using sql commands, or can be converted into XML, HTML or Excel. Since we often do not have access to your data source, this is best used in your test lab. You can then share the database file or converted file with us. The nice thing of recorder is that it records a lot more that the user data: also meta information such as time stamps are recorded.

Tools to analyzer the state or configuration of the system

8. We use  RTI Analyzer when we suspect a configuration issue: e.g. a quality of service (QoS) mismatch. Another good use case is when you have typo in the topicname. DDS will consider this a different topic. Analyzer will quickly show you all the topics in your system, as well as the qos parameters used by the different DDS entities (reader, writer, publisher, subscriber). The match analysis shows you in one screen which reader/writers are compatible.

One common misunderstanding is that Analyzer can not show you all quality of service paramters of a specific entity. For example resource limits or batching configuration are not shown. This is because those are local configuration parameters and are not sent out during discovery. If the configuration is not sent out during discovery, RTI Analyzer does not know about it.

If you want to want to verify whether you configured your batching correctly, I recommend to use RTI Protocol Analyzer as it will distinguish between regular data and batched data samples. It will even show the indivual batched samples.