Among the support team, when we learn about a new debugging tool, we frequently joke about ‘our network debugging tool belt’. Over the years, this tool belt has grown to a nice variety of debugging tools, especially when you include platform specific debugging tools. We currently support a great variety of platforms: from the enterprise platforms (Linux, Solaris, Windows) to various embedded platforms, (including VxWorks, LynxOS and Integrity).

Below is a list of the tools we frequently use on support. I am leaving out platform specific tools, tools to monitor how a particular network stack is processing network traffic (e.g netstat) and general operating system debugging tools (vmstat, cpustat, lsof, gdb, strace etc.).

One of the most common questions on support is: why are my two DDS applications not communicating? The answer to this question falls into two categories: it is either a general networking / connectivity problem, or it is a configuration problem. Let’s take a look at the various tools we use to debug.

Tools to verify connectivity
1. ping verifies whether you have basic connectivity between two hosts. Not commonly known is that ping can also be used to generate multicast data. This is especially useful when your DDS discovery is using multicast and you are suspecting a multicast or time-to-live (ttl) problem.

For example, you can try to ping 224.0.0.1 which is the all-hosts group. If you ping that group, all multicast capable hosts on the network should answer, as every multicast capable host must join that group at start-up on all it’s multicast capable interfaces. There is one gotcha, by default when running a Linux kernel 2.6, the capability of ICMP to reply to broadcast/multicast ping message is disabled. Whereas, it is enabled by default in Linux kernel 2.4. You can enable it by disabling icmp_echo_ignore_broadcasts (reference: http://kerneltrap.org/node/16225)

If you want to experiment with the time-to-live value many ping implementations allow you to configure this via -t or -T.

2. traceroute (unix) or tracert (Windows) is a tool to visualize packet path used when sending a packet. It shows you the different hops / routers a packet traverses.

3. rtiddsping is a command line utility found in the scripts directory of your RTI Data Distribution Service installation. It basically does the same thing as ping, but not by using the ICMP protocol, but by using DDS. If you were able to ping the hosts and wonder whether the issue is a problem in how your DDS application is configured, then rtiddsping is a neutral application to verify whether the DDS ping /pong topic is working. Take a look at the various flags to modify the peerlist, ttl, multicast and reliability qos parameters.

Tools to analyze the data on the wire
4. One of my favorite network tools is probably Wireshark. Wireshark, previously called Ethereal, is a free packet sniffer and allows you to inspect the data on the wire. It will decode the packets based upon various protocol dissectors. The DDS wire protocol is commonly known as RTPS – real-time publish subscribe wire protocol. RTPS dissectors in earlier versions of Wireshark supported an early non-standard version of the protocol. RTPS2 is the dissector which supports the OMG standard of the wire protocol. RTI contributed the source of this dissector to the Wireshark community. We also offer it as part of our RTI Protocol Analyzer product. If you are using RTI Data Distribution Service 4.2 or later, use the RTPS2 dissector.

We use RTI Protocol Analyzer to debug discovery misconfiguration. You can observe which IP addresses are used during discovery announcements or which ports a particular node is contacting. You can also look at whether the reliability protocol needs to repair samples and how many. You are able to filter on Hearthbeats as well as corresponding ACK/NACK packets. Lastly the nice thing about it is that you can save it to a file to share with the RTI support team. We can partake in the analysis of why something isn’t working as expected.

Make sure you are monitoring the data both at the send and receive side. RTI Protocol Analyzer can only capture what is on its wire. If you can not install RTI Protocol Analyzer on the publisher or subscriber, hooking up a simple hub is a good option. A switch, unless you enable port mirroring, won’t do the job. (Doh, where did I leave my old hub now?)

5. snoop (Solaris) is a packet sniffer for Solaris. Unfortunately it does not dissect the RTPS protocol.

6. rtiddsspy is a command line utility, found in the scripts directory of your RTI Data Distribution Service installation, which allows you to automatically subscribe to publications in a specific DDS domain. You can observe both the discovery data as well as the user data. Take a look at some of the command line options to filter out particular topics.

7. RTI Recorder is a more complete data recorder. It can be configured to record specific topics and fields to a database which lateron can be queried using sql commands, or can be converted into XML, HTML or Excel. Since we often do not have access to your data source, this is best used in your test lab. You can then share the database file or converted file with us. The nice thing of recorder is that it records a lot more that the user data: also meta information such as time stamps are recorded.

Tools to analyzer the state or configuration of the system

8. We use  RTI Analyzer when we suspect a configuration issue: e.g. a quality of service (QoS) mismatch. Another good use case is when you have typo in the topicname. DDS will consider this a different topic. Analyzer will quickly show you all the topics in your system, as well as the qos parameters used by the different DDS entities (reader, writer, publisher, subscriber). The match analysis shows you in one screen which reader/writers are compatible.

One common misunderstanding is that Analyzer can not show you all quality of service paramters of a specific entity. For example resource limits or batching configuration are not shown. This is because those are local configuration parameters and are not sent out during discovery. If the configuration is not sent out during discovery, RTI Analyzer does not know about it.

If you want to want to verify whether you configured your batching correctly, I recommend to use RTI Protocol Analyzer as it will distinguish between regular data and batched data samples. It will even show the indivual batched samples.

RTI Data Distribution Service provides a great amount of information about the status of your real-time distributed system. Being able to collect and log this information is essential to start debugging a DDS based application.  There are several ways to implement a logging system which has minimal impact on performance and does not require a recompile. Before we get to that, let’s first take a look at what information is available and what will typically be requested by the support engineer.

Configuration Information
The RTI middleware is configured through various Quality of Service (QoS) parameters.  QoS parameters can be specified for different DDS entities: domain participant, publisher/subscriber, and writer/reader. Easy access to the QoS configuration of each DDS entity within an application is a must. At a minimum the list should include QoS parameters which have been changed from the default values. When using version 4.3 or later of RTI’s middleware QoS parameters can be loaded from an XML file. The XML file is an easy way to change the QoS parameters at start up time, but also to share the configuration.

Part of how each DDS application is configured is how it will discover other applications and matching DDS topics. Gathering the following discovery information is key:

1. What is my initial peer list of each domain participant, used to initially announce myself to other applications?
2. What is multicast receive address used by various domain participants? Is multicast enabled in my DDS application?
3. What transport are enabled (See: DDS_TransportBuiltinKindMask) and what are the transport properties used by each domain participant? These properties configure the UDP and shared memory transports at a low level.

Error logging and status information
By default RTI Data Distribution Service  will log error messages using the NDDSConfigLogger class to the console. The default verbosity configuration is NDDS_CONFIG_LOG_VERBOSITY_ERROR. We encourage you to log this information to a file, and to allow the verbosity level to be modified either at run time or at start up to include warnings (NDDS_CONFIG_LOG_VERBOSITY_WARNING ) or overall status information (NDDS_CONFIG_LOG_VERBOSITY_STATUS_ALL).

DDS status information – DDS data readers and writers maintain status information which can be queried or can be made available through corresponding callback routines when a change in status occurs. This status information provides a great deal of insight into each entity. Not all of these status fields are indicative of an error. However they will help in debugging a problem or a mis-configuration. It is strongly recommended to instrument the corresponding callbacks and log changes to the following data reader and data writer status fields:

· DDS Data Reader status info includes:  DDS_LIVELINESS_CHANGED_STATUS, DDS_REQUESTED_DEADLINE_MISSED_STATUS, DDS_REQUESTED_INCOMPATIBLE_QOS_STATUS, DDS_SAMPLE_LOST_STATUS, DDS_SAMPLE_REJECTED_STATUS, DDS_SUBSCRIPTION_MATCHED_STATUS

· DDS Data Writer status info includes: DDS_LIVELINESS_LOST_STATUS, DDS_OFFERED_DEADLINE_MISSED_STATUS, DDS_OFFERED_INCOMPATIBLE_QOS_STATUS, DDS_PUBLICATION_MATCHED_STATUS, DDS_RELIABLE_READER_ACTIVITY_CHANGED_STATUS

The DDS Data Writer DDS_RELIABLE_WRITER_CACHE_CHANGED_STATUS provides a summary of the state of a data writer’s cache of unacknowledged samples written. Logging this continuously will likely have an impact on performance as the callback will be triggered each time all samples have been acknowledged. We recommend to implement the ability to turn this on at start up or at run time as needed.

DDS Data Reader and Writer Cache and Protocol status – Version 4.3e and later adds statistics information about what is going on at the protocol level. Making this information selectively available will speed up isolation of the problem. The protocol status provides information such as the number of samples pushed, filtered and the status of wire protocol traffic. The cache status provides information about the queue: e.g. number of samples in the reader or writer queue.

For examples or tips on how to implement the recommended guidelines above, contact RTI support or visit the RTI Customer Portal solution section.

Obviously there is RT… uh … the documentation. The doc directory as well as the documents section of the RTI Customer Portal (http://www.rti.com/support) includes both the API (in html and pdf) as well as a comprehensive user manual. The nice thing about the RTI Customer portal is that it also allows you to search the API: by version and by language binding. If you colleague advises you to modify the max_heartbeat_retries to tune the reliability protocol, you can quickly figure out that in C++ you need to modify the DDS_RtpsReliableWriterProtocol_t structure.

In the recent release of RTI Data Distribution Service, the Getting Started Guide has been revamped to make the introductory experience more pleasant. The Getting Started Guide not only walks you through a basic example, but quickly introduces you to some common design patterns: how to configure the middleware for reliable messaging, how to achieve high throughput for streaming data and lastly how to send data over unreliable network connections.

Secondly, a great resource for frequently asked questions is our solutions database. Both the public knowledge base (https://www.rti.com/kb/index.html), as well as our more comprehensive solutions section of the RTI Customer Portal, include various code examples (in C, C++, Java) on important DDS concepts: keys, how to use QoS profiles, built-in and custom content filters, typecodes, etc. You can also find answers to questions such as “What’s the maximum message size supported by RTI Data Distribution Service”, and operating specific tips on how to tune your platform for better performance.

Take a look and let us know what is missing. We continuously add more items to this list. Contact us at support@rti.com.