We’ve Got the Golden Ticket! Reply

chocolate-factory-tile Milk chocolate, dark chocolate, white chocolate…Golden tickets? Yes. The newest Case + Code example is probably the tastiest that we’ve built so far. This DDS use case example illustrates how to use RTI Connext DDS to build a chocolate factory. More specifically, this example shows a chocolate batch that is acted on by a series of station controllers according to a recipe. The batch moves along the workflow, and the station controllers add the right ingredients to turn the batch into milk, dark, or white chocolate. A manufacturing execution system (MES) controls the process by deciding how many batches to produce, and whether each batch will be dark, milk, or white chocolate. The MES also monitors the progress of each batch as it is processed by the station controllers. Delicious. BlogPost-ChocolateFactory This use case can apply to more than just a chocolate factory, of course: this use case represents a class of problems where a series of applications or controllers must process an object, and update its state as they process it. These applications are usually described as discrete processing or workflow applications, and they include a broad range of industries, including:

  • Chemical batch processing
  • Pharmaceutical manufacturing
  • Medical sample processing
  • Food processing
  • Automobile manufacturing
  • Other factory and automation systems

Why does RTI Connext DDS make sense in this type of application? Increasingly, manufacturing systems need to be reconfigurable to adjust to changing market needs, including changing demand, competition, and regulation. RTI Connext DDS provides a platform for implementing a flexible, expandable bus that meets the high-performance requirements of applications on the factory floor. This flexible, near-real-time Manufacturing Service Bus (MSB) allows the developers of manufacturing systems to:

  • Easily reconfigure systems: RTI Connext DDS provides dynamic discovery that allows applications to communicate seamlessly after being restarted or moved to a new network location.
  • Update existing applications: RTI Connext DDS provides extensible data types that are well-defined but can be modified over time within pre-defined limits
  • Add new controllers, applications, and functionality: The dynamic discovery protocol also allows new applications to join the network seamlessly
  • Remove unnecessary functionality: Non-essential services can be removed without affecting existing services.
  • Monitor application performance: RTI Connext DDS provides distributed logging capabilities that can be used to detect system warnings and faults. RTI also provides tools for monitoring application performance when sending data over the network.

MSB RTI Connext DDS provides the bottom layer of this Manufacturing Service Bus, providing the functionality to dispatch orders, send updates about object states, send and receive alarms, monitor the state of applications or devices, and to secure the communications within the factory floor. This allows devices to be discovered, monitored, and controlled with minimal changes to the system as a whole. This functionality allows manufacturers to become more agile, and to be flexible to changing market conditions. If you would like to learn more, view the Case + Code example to read more details about how the applications represent and update the state of each chocolate batch, how station controllers correlate the recipes from a recipe generator with the commands from the MES, and how the MES monitors the batches as they go through the workflow. If you have questions, or if you like the example, feel free to join the RTI Community to ask or answer questions in our forum. If you really like the example, donations of dark chocolate are happily accepted.

Reactive Programming using RTI Connext DDS and Microsoft Rx 1

Reactive Programming is perhaps among the top few technologies rocking the dev world recently. See ReactConf, InfoQ, and Gartner Hype Cycle. The titans of software technology are pushing reactive programming for mobile and cloud applications. I think this is simply an expansion–a major one–of proven techniques into new classes of applications. I.e., the applications joining the club are new but the principles are really not.

What does it really mean to be reactive? (see video) Traditionally, reactive systems are long running systems that must respond to the external stimuli at speeds determined by its environment. The Reactive Manifesto describes four essential qualities of reactive systems: event-driven (i.e., push messages to consumers), scalable (i.e., accommodate dynamic load while maximizing resource usage), resilient (i.e., isolate faults), and responsive (i.e., high degree of predictability).

It may or may not surprise you but RTI Connext DDS rocks in the reactive world. DDS is an excellent choice to build distributed reactive systems because it provides the necessary building blocks:

  1. Event-Driven: DDS is an event-based publish-subscribe middleware and it is asynchronous.
  2. Scalable: Peer-to-peer architecture of RTI Connext DDS scales
  3. Resilient: DDS allows very loosely coupled components which helps isolate faults when then occur making systems resilient.
  4. Responsive: DDS is designed for real-time systems that often have very stringent timeliness requirements. The utility of DDS quality-of-service configuration could not be overstated here.

So, what’s the point? … Let me ask you this …

What’s common between spreadsheet cell formulas and a factory automation system?

Well, they are both reactive! Cells react to changes in the data and robots react to rolling stuff on an assembly line. The difference, however, is how they are built. Spreadsheet formulas are simply declarations of the intent: $(C3)=$(A1)+$(B2).  Most DDS-based systems including factory automation systems are far from that because of the tyranny of imperative programming models and callback-hell–a (perhaps familiar) program structure where asynchronous callbacks, state-machines, and synchronization conspire against you.

Wouldn’t it be nice to be able to develop DDS systems that interact with moving, turning, rotating, and flying things in a much clean, succinct, declarative way that does not degrade in to callback-hell? … Is that even possible?

Indeed.

Rx4DDSRTI Research is excited to announce the Rx4DDS.NET library that integrates Microsoft Reactive Extensions (Rx) with RTI Connext DDS on the .NET platform. Rx and DDS are quite complementary because Rx is based on the Subject-Observer pattern, which is analogous to the publish-subscribe pattern of DDS. Furthermore, the core tenet of Rx composition of operations over values that change over time complements DDS instances, which are data objects that change over time. DDS ensures propagation of changes to the interested remote participants. Consequently, combining Rx with DDS enables a coherent end-to-end dataflow architecture for both data distribution (which is performed by DDS) and processing (which is done by Rx). Rx and DDS together support location transparency of dataflow-style programs seamlessly. The resulting applications dramatically simplify concurrency to the extent that it can be simply configured.

So how does a reactive program using Rx4DDS.NET really looks like? Here is a swap program in C#.

swap

The above program subscribes to the “Square” topic, swaps original x & y coordinates, and publishes triangles of the same size/color (but swapped x & y) on the “Triangle” topic. The little arrow (=>) that looks like a parameter to Select, defines a lambda (an anonymous function) with a single parameter names shape and the part after the arrow is the body of the lambda function. Btw, it also reacts to disposed “Square” and disposes the blue triangle in response.

Complex Event Processing

With Rx4DDS.NET, Complex Event Processing (CEP) is at your fingertips. Thanks to LINQ (Language INtegrated Query). CEP fans among you might be thrilled to see the following example.

correlate

That’s right. With Rx4DDS.NET, you can write SQL-like queries on streaming DDS data. You may think of it as a “continuous query” over streaming DDS data. The above query correlates squares and circles of the same color and produces triangles at the same location and color as that of the squares. The size of the triangles, however, changes with the x position of the circle. This example also uses a thread-pool (Scheduler.Default).

So, what’s the magic sauce here? It is called Functional Reactive Programming (FRP). FRP is a declarative approach to system development wherein program specification amounts to “what” as opposed “how”. The reason above programs are rather short is that FRP offers high-level abstractions that avoid the verbosity that is commonly observed in callback-based techniques. It allows chaining of “operations” on streaming data.

There is a lot more to Rx4DDS.NET than what you see here. In particular, it is designed to support keyed topics, discovery events, communication status messages, and you can correlate all of them reactively and concurrently. So download Rx4DDS.NET and Go Reactive today!

If you are interested in learning about how RTI and Vanderbilt University implemented real-time analytics for a soccer game using Rx4DDS.NET, see this paper and the rticonnextdds-reactive repository that includes all the sources.

Code Generator 2: Generate Code Faster Reply

Are you tired of waiting for all your code to be generated? Would you like to customize the generated output? Code Generator 2 is your solution. This new code generator is already available in your RTI Connext DDS 5.1.0 installation as an Early Access Release (EAR).

Code Generator 2 is written completely in Java and uses Apache Velocity (VLT) templates to define the generated output. This allows you to customize the generated code for different languages by modifying those templates. Check the predefined set of variables available in RTI_rtiddsgen_template_variables.xlsx, located in the Code Generator 2 documentation.

This new design also improves performance compared to the current code generator. The VLT templates are parsed faster than the XSLT ones (used by the current code generator). When generating code for a large IDL, Code Generator 2 is up to 10 times faster than the current code generator.

If you want to invoke the code generator multiples times with different parameters or IDL files, Code Generator 2 provides a server mode that further improves its performance. When used in server mode, Code Generator 2 runs as a native executable that connects through TCP with a server instance of the code generator. The server instance is initiated automatically the first time that the code generator is invoked and it is automatically stopped after an inactivity period. As a result, the JVM is only loaded once when the server is started, the VLT templates are compiled once and the following invocations to Code Generator 2 are resolved faster.

BoostingPerformance2.jpg

Do you want to try Code Generator 2? Just run the rtiddsgen2 or the rtiddsgen2_server (for running in server mode) scripts with the usual parameters. If you need more information about supported and new features available in Code Generator 2, check out its Getting Started Guide.

RTI RefleX: Reflection-based Type Modeling for DDS Topics Reply

Data-Centricity–a design approach that places the data first and foremost in the thinking, construction, and operation of a system–is perhaps the most salient feature of DDS. Data-centric systems are everywhere: databases, for one, REST is another. Similar to databases, DDS is aware of the structure of the data used by the applications to communicate. Naturally, DDS needs a way specify the structure.

OMG Extensible and Dynamic Topic Types (i.e., DDS-XTypes) specification defines rules, actually, a type system, that governs the structure and behavior of DDS topic types. These rules are are very similar to that of C++ and Java. Concretely, DDS-XTypes provides four different ways to specify types: Interface Definition Language (IDL), XML Schema Definition (XSD), XML, and the TypeObject API. The first three are declarative ways to specify types (think C++ structs), whereas the last one is an API, which means the types can be created and observed at runtime.

To make things just a little bit more complicated, the OMG Java 5 PSM for DDS allows Java types as a way to specify DDS topic types. In fact, any class that implements java.io.Serializable interface is available for publishing and/or subscribing over DDS. As Java types can be introspected using the Java Reflection API, it is relatively straight-forward to map a Java type to a DDS-XTypes compliant type using the TypeObject API mentioned before.

But, alas, no comparable support exists for C++. The OMG C++ PSM for DDS makes no provision to use native C++ types in a similar fashion. Perhaps because C++ has no standard reflection API . So, does that mean C++ programmers have to lament in solitude forever?

Nope!

RTI research is happy to announce the RTI Connext DDS RefleX library for C++. RefleX is a short for Reflection for DDS-XTypes. The crux of this library is to create XTypes compliant type representations directly from native C++ types. RefleX is declarative–i.e., it is not a reflection API. There is no separate code generation step involved. The RefleX library will take your application-level datatypes, no matter how complex, and will happily map them to equivalent DDS topic types.

To do that, it needs just a little help from you. It needs you to specify what’s in your type. Let’s take an example. Around here, that means Shapes! A C++ native ShapeType struct is defined on the left. To start using it with RTI Connext DDS, all you have to do is to specify what’s on the right hand side. And that’s it. You can start using ShapeType with GenericDataReader and GenericDataWriter (also provided by the RefleX library) as follows.You can think of RTI_ADAPT_STRUCT as a substitute for the lack of reflection in C++. It is not the kind of reflection in the Java/C# world. It is a macro, which opens the guts of the specified type to the RefleX library. The key difference is that the process of “reflection” happens at compile-time. The Reflex library uses powerful C++ meta-programming (templates and the preprocessor) machinery to iterate over the members of the user-defined type and synthesizes equivalent TypeObject representation for the type and DynamicData representation for the state. The resulting on-the-wire type description is fully compatible with an equivalent type described in IDL. Needless to say, they interoperate.

The compile-time translation is a two step process:

  1. First, a TypeObject representation is generated using the TypeCode API. This ensures that a data-centric representation is preserved during translation.

  2. Second, the TypeCode is used to create an empty DynamicData object. It is later on populated with a user-supplied native C++ object.

Note that there is more than just the name and the type of the members. A TypeObject representation has other ornaments, such as bounds, keys, optionality, whether a member is shareable or not, and extensibility.

Performance wise, compile-time reflection has zero cost because there are no run-time calls to find out what’s in a type. RefleX knows that already through the RTI_ADAPT_STRUCT macro. The only necessary cost incurred is due to the creation of a DynamicData object for each sample. RTI’s DynamicData implementation is highly optimized for depth-first member access with increasing member IDs. RefleX traverses types in depth-first fashion and therefore, by the time it is done traversing a native C++ object, the serialized representation of the object is ready to be sent over the network.

So how complex the native type is allowed to be? … Very complex!

Supported types in the C++ point of view

Supported types in the XTypes type system point of view

  • Fundamental types, arrays, enumerations

  • Struct (with public members)

  • Classes (with setter/getters)

  • Nested struct/classses

  • Standard Template Library (STL) string, vector, list, set, map, array, tuple, pair, iterators, etc.

  • Smart pointers

  • User-defined/custom containers

  • Lazy container adapters (e.g., boost.Iterators)

  • Most combinations of the above

  • Basic types/enumerations/strings

  • Arrays

  • Sequences of strings/basic types/enumerations/structured types

  • Bounded sequences/strings

  • Structures

  • Unions (including cases with defaults, multiple discriminators, and enumerations)

  • Optional members (sparse types)

  • Sequences of sequences (of sequences… and so on…)

  • Multidimensional arrays of strings, structures, sequences,…

  • Nested structures, unions

The two columns above specify two different things: (1) What native C++ types can be directly mapped to DDS-XTypes and (2) What C++ type is needed to match a given IDL type. The later also means that discriminated unions can be created using native C++! However, such cases are rather unusual and it is perhaps better to use IDL-based code generator.

Relationship to the OMG DDS C++ PSM

The new OMG DDS C++ PSM is independent of RefleX. They are complimentary. In most cases, the PSM requires rtiddsgen generated types, which will in turn use C++ STL containers (vector, map, string, etc.). As DataReader and DataWriter are parameterized using the generated types, the programming experience is similar to that of RefleX. So with the new C++ PSM, the need for RefleX may be reduced but not eliminated. See the advantages of RefleX below.

Advantages of RefleX

  • Native Types (IDL-free): IDL representation of DDS types is no longer necessary. Native C++ types declarations in the headers is sufficient. With RefleX you are free to use shared-library types, such as STL containers, custom iterators, shared_ptr, auto_ptr, unique_ptr, etc. The RefleX library has a predefined mapping for each and you can also define your own mapping. More on that later.

  • Declarative Dynamic Data: DynamicData and TypeCodes objects can be created declaratively as opposed to procedurally, which quickly gets laborious and repetitive as the complexity of the type increases. Many times, types are known statically but no code generation support is available (e.g., RTI Routing Service adapters). With RefleX, creating a TypeCode and populating a DynamicData object is never more than one line of code!

  • Third-party code generators: Classes generated by third-party code generation tools can be mapped to DDS topic types. The key is that RefleX works directly off of C++ types. The generated types may use a wide variety of types (e.g., boost containers, custom iterators, etc.). RefleX provides a way to map such types directly to a DDS. For example, with RefleX the types generated by a third-party XSD to C++ compiler can be mapped to DDS adding an RTI_ADAPT_STRUCT macro for each generated type.

  • Glue code: Glue code must often be written because wire-level types may not always match with the application-level types. RefleX allows direct use of application-level types with DDS. Glue code is nearly eliminated.

  • Performance: Glue code implies copying of data back and forth between application-level data types and the idl data types. Copies are not required while using RefleX. Note, however, that marshaling and demarshling of is still necessary.

The RefleX library is now available for download from the RTI Community github repository. There are several examples included in the sources. To compile the library you will need a fairly good C++ compiler. At the time of this writing, gcc 4.8, clang 3.3, and Visual Studio 2013 are supported.

More reading? Check out whitepaper on RefleX and a presentation on Slideshare.

Recording Service 5.1: Faster, More Scalable and More Concurrent than Ever! Reply

You’re facing a problem in your DDS system: you want to use Recording Service 5.0 to record high throughput data coming in from sensor networks. The database has to be accessed by other applications while Recorder is continuously recording. Your topics are updated frequently and Recorder has to write at such speeds that it locks some of your other applications out of the database. When you open the database, you realize that although your types are small, the tables are big and full of columns you don’t really need. What can you do??

Good news!

RTI Recording Service 5.1 was released a month ago with one of the biggest feature releases across the entire suite, including Recorder, Replay, Recording Console and Converter.

There were three motivations behind the updates to Recording Service 5.1 (RS):

  1. The OMG XTypes Specification, which required integration with XTypes, including Mutability and types with optional members.

  2. Enhanced scalability and performance, which resulted in additional configurability of the SampleInfo and metadata fields being recorded, and a reduction in new default column sets.

  3. Improvements in data concurrency, especially in systems where the database is accessed by other applications while Recorder is recording.

Your types evolve? No problem. Recorder records them, Replay replays them

Systems evolve and as they evolve, so do their data and data types. Recorder and Replay support the XTypes specification, with mutable types and types with optional members. Recorder and Replay allow you to choose what version of a type to record or replay, by providing the types via XML. A new feature that easily allows mapping of topic names and type versions in the configurations.

RecordingService5.1BlogPost-1(1).png

Recorder and Replay administration types and topics are now also using X-Types. From this point forward (starting with 5.1), if the types evolve in the future, legacy applications will be able to administer the tools.

Configurability for enhanced scalability and performance

One of the coolest new features in Recorder is the ability for the user to select the SampleInfo and metadata fields to be included in user topic tables, as well as which fields to store in the discovery tables. Also now, by default, only a few extra fields are recorded apart from those of the user data types – just the necessary fields for Replay and Converter to work with the database file. See section 4.5.1 of Recording Service User’s Manual for details and examples.

These new settings and feature enable a boost in the performance and scalability of Recorder. It can now record more user data fields and fewer irrelevant fields. Compared with version 5.0, the size of the database can be significantly reduced in cases where the user types are small- to medium-sized.

Boosting concurrent access to the database while Recorder is recording

There are new features that enable improved concurrent access to the database while it’s being recorded. Highlights include:

  • Ability to specify SQLite pragmas upon creation of the database file (and subsequent segments).
    Among other things, this gives the ability to change SQLite from journal mode to WAL mode. Write-Ahead Logging offers improved concurrency because readers of the database don’t block writers and a writer doesn’t block readers.

  • Online indexing.
    Recorder can create and maintain indexes on the database while recording. Indexes can be established for any SampleInfo fields or for user data fields (when recording in deserialized format). When other applications need efficient access to certain tables, and the access is based on the content in certain fields in those tables, online indexing can really improve the overall performance of the system. However, remember there is a tradeoff: Recorder’s performance will decrease as the number of indexes to maintain increases. Check out section 4.8.2 of our RTI Recording Service User’s Manual 5.1 for tips and tricks for indexing in SQLite databases.

And more…

There are lots of new features, improvements and cool changes I haven’t covered here, including automatic path separator detection in Replay, a new and faster deserialization algorithm for Converter and topic filters in QoS settings.

Check out the RTI Recording Service 5.1 Release Notes to get a sense of all this new stuff!

30dayFree

RTI Challenges Tomorrow’s Engineers Reply

RTI and the University of Granada have a long-lasting relationship. Not only have several of our engineers studied there, but we are also actively collaborating with the University on research and educational projects.

One of the more interesting projects we collaborate on is the Technical Challenge (Desafío Tecnológico). This Challenge is a competition between several groups of students to develop a product and “sell” it to some investors, i.e. the jury of the competition.

This year, RTI proposed a multiagent-based distributed video system as the goal for the competitors. The competitors need to develop a system that provides reliable video and can receive new agents on the fly. We expect they’ll take advantage of the reliability and scalability capabilities of RTI Connext DDS and build something like this:

techChallengeDiagram

RTI wants the students to succeed and try building a real world application. We are providing them with Raspberry Pi boards so they can set up their agents realistically and present a good prototype to the Challenge jury.

raspberryPi

The Challenge is a year-long project that started in October 2013. We will see the final products this fall, but you can get a little more information from this 2-minute video:

We hope the Technical Challenge becomes an annual event. Check back to see the winners and try to sign up next year!

Features and Enhancements and Support – Oh My! Reply

connext dds 5.1.0_wordCloud

As engineers we all know that product releases are no joke. The making of a great product is fueled by hard work, and lots of it. It takes teams of people working together to make something bigger and better than any one of them could do alone. The results generated by all of that hard work can be downright inspiring and impressive. But do you know what my favorite part of the release is? It’s the feeling when we ship it to the public. When all of the ducks are in order. Every person in the company, irrespective of their group, has contributed something necessary to getting the product to that point. Seeing it all come together? That’s my favorite part. I love it.

Did I mention that RTI Connext DDS 5.1.0 was released 2 weeks ago? It was!

To help ensure your transition to the new version is as seamless as possible, as well as keep you informed on the new features and enhancements you’ll see in version 5.1.0, I thought a post that served as a cheat-sheet of sorts might be helpful. We have put a lot into this release and I’m sure that you’ll love some of the great new features such as our new Built-in QoS Profiles and additional support for DDS-XTypes . For up-to-date product information and more information about the release, don’t forget to visit our Community Portal follow us on twitter .

I hope you enjoy 5.1.0. We certainly enjoyed all of the work we did bringing it to you.

Connext DDS 5.1 – Download. 

downloadButton

Community Newsletter – Special Edition. A Special Edition newsletter, focused solely on the 5.1 release, was sent out to all registered community members. You can find a copy of it here (although, you all should be registered community members so you would have already received and read this!).

Free Webinar + Slides/Replay Available.

photo (2)

David (left) and Jan (standing,right) during their webinar on February 27th.

David (our VP of PM) and Jan (VP of Engineering) presented a webinar where they dove into the new release and it’s capabilities – it was pretty awesome. If you missed out, don’t worry! You can view the replay (here), the slides (here), and if you have any questions about the webinar content – or the release! – you can post them on the Community Portal forum. We’d love to hear from you and help you out!

5.1 Blogs. A handful of posts were put together to inform you of some new features and capabilities in this release.

Additional content on the Community Portal.

Also be sure to check out the Connext DDS landing page on our main website. It’s been updated with all kinds of info on the release, including a video.

Build a Multi-Location Temperature-Monitoring System using Connext DDS & RTI Prototyper with Lua 1

Some employes in the northwest corner were feeling colder than the engineers on the south side of the building. The engineers, of course, thought it was simply because they were hotter than anybody else, but they took up the challenge to discover whether there was really a problem. In fact, we attempted to use our own RTI Connext DDS and a bunch of other technologies to prove how easy it is to integrate different systems together.

More…

Filter Propagation: Because Efficiency is Golden 1

Considering the fact that limited resources are simply a reality, learning to be efficient is key. Unfortunately, we run into many situations in our daily lives where resources are wasted due to poor utilization.

We throw away tons of advertising papers from our mailbox just because we didn’t ask for them. But what if we had a mechanism to specify whether we were interested in specific advertising? Then, only the exact amount of resources – paper, manufacturing, delivering – would be consumed.

RTI began supporting this premise long ago with the addition of content filters. Subscribers use them to select which information they are interested in, and with writer-side filtering, publishers only send selected information, saving 3 precious resources: CPU utilization, memory and bandwidth.

More…

XTypes: Taking Type Evolution To The Next Level 3

RTI Connext DDS 5.0.0 provided limited support for type evolution by implementing Extensible type extensibility as defined in the “Extensible and Dynamic Topic Types for DDS” specification, commonly referred to as the DDS-XTypes spec.

With Extensible type extensibility, the types in a DDS system can evolve by adding new elements at the end.

 For example, let’s assume a GPS vehicle tracking system. In a first implementation, the VehicleData type may provide position information such as latitude and longitude:

More…