<img height="1" width="1" src="https://www.facebook.com/tr?id=1357777224302944&amp;ev=PageView &amp;noscript=1">

Useful Tools to Debug DDS Issues

June 15, 2017 by
|

While developing an application using RTI Connext, DDS users may run into a situation where the publisher and subscriber are not communicating. In these situations, we will usually get the question: how can I figure out what the issue is and how to solve it?

There are several tools and features that can help you debug your DDS issues:

1. Look for log warnings or error messages.

You can enable log messages in your DDS applications. There are different verbosity levels depending on the kind of messages that you would like to see. Usually, we recommend that customers use the  “Warning” verbosity level to see the most common problems. If you need further details in a certain situation, you can increase the verbosity level to “All.” However, be aware that setting the highest verbosity level will result in the output of lots of messages.

You can set logging via XML or code. To set it via XML – with warning verbosity and sending the output to a file – you need to add the following settings to your XML QoS profile:

         <participant_factory_qos>

           <logging>

             <output_file>log.txt</output_file>

             <verbosity>WARNING</verbosity>

             <print_format>TIMESTAMPED</print_format>

           </logging>

         </participant_factory_qos>

To enable logging via code, add the following lines to your application main function (C++ example):

         NDDSConfigLogger::get_instance()->

         set_verbosity_by_category(NDDS_CONFIG_LOG_CATEGORY_ALL,

         NDDS_CONFIG_LOG_VERBOSITY_WARNING);

      FILE *myLogFile = fopen("my_logfile.txt", "a+");

      NDDSConfigLogger::get_instance()->set_output_file(myLogFile);

Note that this code needs to be added before you call any other operations on the RTI Connext DDS API to ensure that any output goes to the file instead of the standard output.

If you need to limit the size of the log output file, follow the steps in this KB.

RTI Log Parser

Once you have the log messages, you can parse them using our RTI Log Parser. This is a command-line tool that processes and enhances Connext DDS log messages, making it easier to debug applications.

Let’s take the following log messages as an example:

COMMENDSrReaderService_onSubmessage:[1484571004,3507893055] reader oid 0x80000004 received DATA of sn(0000000000,00001346), vSn(0000000000,00001346) from writer 0xffffffff.49e6.1.80000003

COMMENDSrReaderService_onSubmessage: accepted sn(0000000000,00001346), dataRcvd.lead(0,1346), nextRelSn(0,1347), reservedCount(1)

COMMENDSrReaderService_onSubmessage:[1484571004,3577888137] reader oid 0x80000004 received DATA of sn(0000000000,00001348), vSn(0000000000,00001348) from writer 0xffffffff.49e6.1.80000003

COMMENDSrReaderService_onSubmessage: accepted sn(0000000000,00001348), dataRcvd.lead(0,1347), nextRelSn(0,1347), reservedCount(2)

MIGInterpreter_parse:HEARTBEAT from 0xffffffff,0X49E6

COMMENDSrReaderService_onSubmessage:[1484571004,3578261799] reader oid 0x80000004 received HB for sn (0000000000,00001345)-(0000000000,00001348), epoch(347) from writer 0xffffffff.49e6.1.80000003

COMMENDSrReaderService_sendAckNacks:[1484571004,3578261799] reader oid 0x80000004 sent NACK of bitmap lead(0000000000,00001347), bitcount(2), epoch(353) to writer 0xffffffff.49e6.1.80000003

COMMENDSrReaderService_onSubmessage:[1484571004,3578751425] reader oid 0x80000004 received DATA of sn(0000000000,00001347), vSn(0000000000,00001347) from writer 0xffffffff.49e6.1.80000003

COMMENDSrReaderService_onSubmessage: accepted sn(0000000000,00001347), dataRcvd.lead(0,1347), nextRelSn(0,1349), reservedCount(-1)

MIGInterpreter_parse:HEARTBEAT from 0xffffffff,0X49E6

COMMENDSrReaderService_onSubmessage:[1484571004,3579107907] reader oid 0x80000004 received HB for sn (0000000000,00001347)-(0000000000,00001348), epoch(348) from writer 0xffffffff.49e6.1.80000003

COMMENDSrReaderService_onSubmessage:[1484571004,3579107907] reader oid 0x80000004 sent ACK of bitmap lead(0000000000,00001349), bitcount(0), epoch(354) to writer 0xffffffff.49e6.1.80000003

This is how the output will look like after being parsed by RTI Log Parser:

2017-01-16T12:50:03.919122 |  --->   |     H2.A2.P1     |   R-K_800000   | Received DATA [1346] from writer W-K_800000 (reliable)

2017-01-16T12:50:03.919122 |      |                     |             | *Warning: Missing packet for H2.A2.P1.W-K_800000 to R-K_800000*

2017-01-16T12:50:03.919122 |  --->   |     H2.A2.P1     |   R-K_800000   | Received DATA [1348] from writer W-K_800000 (reliable)

2017-01-16T12:50:03.919122 |  --->   |     H2.A2.P1     |   R-K_800000   | Received HB [347] from writer W-K_800000 for samples in [1345, 1348]

2017-01-16T12:50:03.919122 |   <---  |     H2.A2.P1     |   R-K_800000   | Sent NACK [353] to writer W-K_800000 for 1347 count 2

2017-01-16T12:50:03.919122 |  --->   |     H2.A2.P1     |   R-K_800000   | Received DATA [1347] from writer W-K_800000 (reliable)

2017-01-16T12:50:03.919122 |  --->   |     H2.A2.P1     |   R-K_800000   | Received HB [348] from writer W-K_800000 for samples in [1347, 1348]

2017-01-16T12:50:03.919122 |   <---  |     H2.A2.P1     |   R-K_800000   | Sent ACK [354] to writer W-K_800000 for 1349 count 0

For more information about how to configure DDS logging, please refer to the User’s Manual.

2. Check status of DDS entities

  1. Use RTI Admin Console

    Admin Console is a GUI tool that includes many features to easily debug your DDS applications. For example, with Admin Console you can:

    • Monitor log messages and view your QoS, as explained in this blog post.
    • Automatically perform match analysis of your DDS entities. The match analysis shows if there are any QoS incompatibilities between your DDS entities that prevent them from communicating
    • Visualize your data.                                                                                                            
  2. Enable Monitoring Libraries

    RTI Monitoring Library enables RTI Connext DDS applications to provide monitoring data. The monitoring data can be visualized with RTI Monitor, a separate GUI application that can run on the same host as Monitoring Library or on a different host. Monitoring Library periodically queries the status of all Connext DDS entities and brings up all the entity creation/deletions and QoS changes, among other statistics.

    You can enable the Monitoring Libraries by setting values in the DomainParticipant’s PropertyQosPolicy (via code or through an XML QoS profile). The specific instructions to enable Monitoring Library can be found in the User’s Manual. If you want to configure the behavior and statistics obtained by Monitoring Library, you can refer to this section of the User’s Manual.

  3. Check entity status

  4. DDS entities have different statuses associated with them. The statuses give information about important changes in the entity state. For example, there are statuses that are triggered when a sample is lost or rejected, and also a status to detect the liveliness loss of an entity. An entity status can be retrieved by:

    • Directly checking the status with the appropriate get API
    • Using listeners
    • Using StatusConditions and WaitSets

     This Community Best Practice article explains the difference between these methods.

     You can find the different statutes associated with an entity in the User’s Manual and in the online API documentation, for example for DataWriters and DataReaders.

3. Check what’s happening on the wire

Looking at the traffic generated by your applications can give you a lot of information about what is going on. To do this you can use Wireshark.

  • You can find documentation about how to use Wireshark with RTPS messages in our documentation section “Wireshark and RTPS.”
  • Wireshark cannot capture traffic from the loopback interface on Windows. You can use RawCap instead to capture this traffic on Windows. Once traffic is captured in a pcap file, you can open it with Wireshark to analyze. These are the steps to use RawCap:
    • Run your applications with shared memory disabled. You can disable shared memory via XML by adding into your <participant_qos> this setting:

           <transport_builtin>

             <mask>UDPv4</mask>

           </transport_builtin>

  • In a terminal run RawCap.exe and select the interface in which you wish to capture traffic.
  • You can configure Wireshark to show RTPS packets with specific colors, as explained in this Community KB.

4. And last, but not least...

You can also find useful tips for system-level basic debugging in this KB article.

Let Us Know What You Thought of the Article.

Leave a Comment Below.

FREE ON-DEMAND WEBINAR

Accelerate Distributed Systems Development using Connext Tools

Connext DDS Tools Tutorial Video
Watch Now

Subscribe to Email Updates