The Future of Medical Reply

We’re working with our customers to share with you their stories and insights, to offer you a rare glimpse into the future of systems from some of the world’s most exciting and innovative industries and development teams. Enjoy!

rti - blog - future of - banner - BKMED

 

michael eibye - BK Medicalbk medical_ultrasound

By Michael Eibye, Manager – R&D Software and System Test, BK Medical

The fundamental premise of a 5-year research program that we at BK Medical are undertaking, focused on next generation ultrasound imaging systems, can be summed up in 6 words: The future of medical is distributed.

As anyone who has ever worked first-hand with large distributed systems can attest, as a distributed system scales, those 6 simple words are something of a Call To Action as well – how do we address the future and ensure we learn from our past experiences, whilst building on the investments customers have already made? Building distributed systems is challenging, but we are confident that this shift will have a far reaching impact on the quality of care systems as they are made more cost effective, while at the same time providing greater integration into the patient care systems, facilitating improved decision making for doctors.

Distributing an ultrasound system requires determining the extent to which the existing stand-alone system (Figure 2) can be re-constructed into constituent elements while maintaining functionality and performance – the transducer for obtaining the data, the algorithmic engine for processing it, the display device for presenting the resultant images, plus the integration with the care environment back office systems. (Figure 1)

Figure 1. Logical Distribution of the BK Medical Ultrasound Scanner

Moving to a distributed solution introduces a new set of challenges: An ultrasound transducer can generate Gbits of data per second. Can this data be safely and securely transmitted over wireless instead of tightly-coupled single system integration? How far can the algorithmic processing engine be separated from the data stream and display device and still deliver timely screen updates? How can we better integrate with hospital systems? Most important of all, how do we migrate these services into our customers and allow them to leverage their legacy systems investments? These are some of the fundamental questions we seek to answer.

There are also commercial drivers behind the shift to distributed: Hardware costs for traditionally tightly integrated system elements, such as the display device, have dropped precipitously in the last few years as tablets have become commonplace. The algorithmic processing power of the PC games engine has driven down compute engine costs and their programmability. In addition the cost of Wi-Fi integration has fallen now that every smartphone has it included, while the tools to enable reliable high performance communication have increased.

Figure 2. State of the Art of a current Ultrasound System

Figure 2. State of the Art of a current Ultrasound System

There are as many drivers towards distributed systems as there are external market pressures: BK Medical is part of a larger organization, Analogic Ultrasound. As we seek to leverage the Intellectual Property and skill base across the companies, we need a development methodology that enables independent development, yet at the same time enables rapid integration of relevant software and hardware IP for new products. With this objective in mind we sought to architect loosely coupled modular software. It became clear the best and most validated approach was to build our systems data-centrically. This decision combined with a need for robust, real-time, scalable software led BK Medical to select DDS (Data Distribution Service) as its software infrastructure.

The core IP of future BK medical products will be software so we are paying great attention to how to construct that software to maximize flexibility. A focus on a decoupled software approach has to work in two dimensions at the same time; De-coupled applications that can be rapidly and easily integrated to bring together new and more powerful system features and functions, plus the software applications and modules have to be decoupled from the underlying hardware and distributed network topology, thus enabling rapid integration of new hardware IP or integration of low cost commercial hardware platforms in new distributed designs. While several software approaches provide the former, few simultaneously enable the latter in distributed systems, especially in future systems that may need real-time communication, can potential scale significantly and have to be very robust. In addition, with an eye to the future of the Industrial Internet of Things and how this will impact the hospital environment, an approach that can be safely secured is also important.

RTI Connext DDS provides us the ideal platform on which to research and develop the future of ultrasound medical systems. After all, at BK Medical we believe that the future of medical ultrasound is distributed.

If you have a story about using Connext DDS that you’d like to share, email us at RTIBlogs@rti.com. 

Building Connext Applications using Android Studio 11

Android Studio is the official IDE for Android application development, based on IntelliJ IDEA. The first stable build was released in December 2014, starting from version 1.0. Android Studio is designed specifically for Android development and it is available for download on Windows, Mac OS X and Linux at http://developer.android.com/tools/studio/index.html. This section will describe how to use Android Studio to build a Connext application. It assumes that Android Studio is correctly installed.

Create an Android Project

This example uses and IDL file to define the types. The IDL file will be created and added once the project s created. The Android App built in this section can interoperate any other RTI Connext application using the same IDL file, same topic name, and compatible QoS settings. If a different IDL file is used, then note that the following steps will need to be altered slightly as the IDL file name, “HelloWorld” is used in completing fields and naming other files in this example. This example shows only the creation of the publisher application. These steps can be repeated to create a HelloWorldSubscriber application. If the subscriber and the publisher will be installed on the same device, make sure the subscriber’s Application, Project, and Package names are different than the publisher’s.

From the Android Studio menu bar, select File -> New Project.

pic1

In the displayed New Project dialog, fill in the Application name, Company domain, and project location. The dialog will select the default project location and uses the application name to create a sub-directory. You can leave the project location and use the default. After you completed all the fields click next.

pic2

 

Select the form factors and click next. For this example we create an Application for a Nexus 7 tablet and selected an API level which makes it compatible with most of the Android devices. You can select different API levels based on your application preferences.

pic3

Choose Blank Activity since we are not creating any UI for this application. For your application you might want to use some of the other templates based on your target application. Click Next.

pic4

Last we select the name for the activity and some other elements. Edit the Activity Name as shown below. There is no need to edit any of the other fields since they are derived from the Activity Name. Click Finish

pic5

After the project has been created, Android Studio will display the Project as shown below. At this point, HelloWorldPublisherActivity.java is the only source file in the project.

pic6

 

Edit the Project Source Code

Edit the generated Java class (HelloWorldPublisherActivity) by adding the two lines shown below. This will create the simplest Android application with Connext. More sophisticated Android code can and probably would be used for a real application. Real Android applications should never call an endless loop from the main activity since this will stop the application from responding.

Android Studio shows HelloWorldPublisher in red text because that class does not yet exist in the project. Save the changes to HelloWorldPublisherActivity.java.

pic7

Create the HelloWorld.idl file in the project’s java folder. Right click on the java folder and select New -> File.

pic8

 

The destination directory dialog box should already show the right location ..\src\main\java. Click OK.

pic9

 

Enter HelloWorld.idl as file name and click OK.

pic10

 

Android Studio will open the newly created file for editing. Add the text as shown below and save the file.

const long HELLODDS_MAX_PAYLOAD_SIZE = 8192;
const long HELLODDS_MAX_STRING_SIZE  =   64;

struct HelloWorld {
    string<HELLODDS_MAX_STRING_SIZE>             prefix;
    long                                         sampleId;
    sequence<octet, HELLODDS_MAX_PAYLOAD_SIZE>   payload;
};

pic11

 

Open a terminal window/command prompt. Before generating the example code make sure that NDDSHOME environment variable is set correctly to the Connext installation directory and that the PATH environment variable includes $NDDSHOME/scripts.

Change the directory to your applications java directory (HelloWorldPublisher\app\src\main\java) and run rtiddsgen as follow to generate the example files

rtiddsgen -language Java -package com.rti.example.helloworldpublisher -example armv7aAndroid2.3gcc4.8jdk HelloWorld.idl

Note that the -package argument is the same value as used when creating the project. If the rtiddsgen-generated java classes are in a different package to that of the project then “import” statements will need to be added to some java files. You can find the package name at the top of the HelloWorldPublisherActivity.java file.

If you run on Windows and do not have the Visual Studio compiler in your PATH environment variable add –ppDisable to the rtiddsgen command line to disable the preprocessor. Since the IDL file doesn’t have any pre-processor statements running the pre-processor is not needed.

pic12

 

You will see the following un your command prompt window:

Running rtiddsgen version 5.1.0, please wait ...
Done

Done means that all the required type support files and an example files have been generated.

In Android Studio you should see all the additional classes which have been generated and the call to HelloWorldPublisher.main in HelloWorldPublisherActivity should now be shown as resolved. The USER_QOS_PROFILES.xml file, the HelloWorldSubsciber class, and the makefile is not used by this project. Those files can be deleted or kept for reference.

 

Add Connext Libraries

Resolve the Connext symbols in the generated java files by adding the Connext libraries to the project.

Copy nddsjava.jar from $NDDSHOME/class to the lib directory of your HelloWorldPublisher App. You should see the library being added as shown below.

pic13

The next step is to add the library to the build. Right click on it and select Add as library.

pic14

The library needs to be added to the application. Click OK.

pic15

If you open the build.gradle file you will see that the library has been added.

pic16

 

Connext is not pure Java, it also consists of some native libraries. When building the project, you will not see any complaints if these libraries are missing, but when you try to run the App, the LogCat panel will show errors such as:

com.rti.example.helloworldpublisher W/System.err﹕ The library libnddsjava.so could not be loaded by Linux.

com.rti.example.helloworldpublisher W/System.err﹕ Make sure that the library is in your LD_LIBRARY_PATH environment variable.

And you will see the following error screen on your Android device

pic17

Below is how I added the native libraries to the Android Studio project.

Add a lib directory to your project with a subdirectory which contains the native Connext DDS libraries (libnddsc.so, libnddscore.so, and libnddsjava.so). for this example named the sub-directory armeabi-v7. This will look as follow in your project.

pic18

Once you have your .so files in that directory configuration, create a .zip file from the lib directory. I named the file lib_rti.zip. The .zip file will have lib/armeabi-v7/*.so as the contents.

pic19

 

Rename the extension of the zip to .jar giving you a file names native-rti-libs.jar.

pic20

 

And then drag the .jar into your Android Studio project libs directory (same location as all your other jar libraries).

pic21

 

Finally, add this to your projects build.gradle:

Compile fileTree(dir: ‘libs’, include: ‘*.jar’)

pic22

 

Update the Android Manifest

Connext expects to use the Internet and Wi-Fi to access the Internet. It needs to be able to change some of the Wi-Fi settings. Connext also needs to access external storage if it is to read a USER_QOS_PROFILES.xml file (although the current example doesn’t make use of that).

In the project Explorer, double-click the AndroidManifest.xml file to edit it. Add the following lines to the manifest file as sown below.

<uses-permission android:name="android.permission.INTERNET"/>
<uses-permission android:name="android.permission.CHANGE_WIFI_STATE"/>
<uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE"/>
<uses-permission android:name="android.permission.CHANGE_WIFI_MULTICAST_STATE"/>

pic23

 

Run the Example

Before running the example make sure you have an Android device connected via USB or have a Virtual Android Device running.

Select Run->Run ‘app’ or click on the green triangle next to app in the tool bar

pic24

Choose the device you want the application to run on. The example below shows a Nexus 7 connected through USB and a virtual device which emulates a Nexus 7.

pic25

Once it is running you will see the output being displayed in the logcat window.

pic26

If you have a machine connected to the same network which has RTI Connext installed you can start rtiddspsy and see the samples being published.

pic27

The sample code doesn’t fill the samples with any actual values. You can add this to your code in the HelloWorldPublisher class. Look for the comment “Modify the instance to be written”

pic28

The example used in here uses a loop to send DDS samples. A real Android application would of course have a UI to control the behavior and not have a loop sending samples with a delay between sending samples. A real Android application should never block in a loop. However I hope this example was useful in showing how to use the RTI Connext libraries and build an application using Android Studio.

The Future of Automotive Reply

We’re working with our customers to share with you their stories and insights, to offer you a rare glimpse into the future of systems from some of the world’s most exciting and innovative industries and development teams. Enjoy!

rti - blog - future of - banner - AUDI

constantin - HIL AUDI AG

By Constantin Brueckner, Hardware-in-the-Loop Functional Test, AUDI AG

Your car is probably the most compute-intensive thing that you own. It will have at least 40-50 Electronic Control Units (ECUs) for a recent economy vehicle and well over 100 for a top of the range car. In the past, each of these ECUs had one dedicated function to perform. This evolved over time and most of the ECUs now perform more than only one single function or group of functions. Despite this evolution of ECU use, there is still an increasing need to reduce the number of ECUs and the cabling between them with the ultimate aim of increasing fuel economy and reducing CO2 emission, whilst providing the customer even greater functionality in the car. These additional demands are being met by a shift towards functional integration and communication between ECUs and between the car and its environment. This is one of many more reasons why the future of automotive test is becoming distributed and interconnected. Furthermore our test systems have to evolve as fast as the car functionality to encompass this change. To address these challenges Audi founded a pre-development department for test systems, which currently develops a real-time capable bus system based on RTI DDS for the future test systems.

But first, let us have a look in more detail at the shift towards ‘functional integration’ and explain it with following examples:

  • A simple former “air bag computer” fired the air bags at the time of a crash. This now becomes an integral element of the complex safety system with more safety functions to avoid great injury to the passengers in case of a crash. The new, so called, ‘safety computer’ has an automatic crash detection capability (“Audi pre-sense”) and it has to perform, for example, fully automated braking support, deploy the air bags, pre-tension the seat belts, close the windows and roof and move seats into an upright position.
  • Dedicated ECUs for radio, navigation and rear seat entertainment are evolving into a “main entertainment unit.”
  • Dedicated ECUs for body electronics like head light, interior light, and air conditioning, are combined into one “body control module” and enriched with new capabilities such as bending light, LED head light, parking assist, air condition and rain-sensing wipers.

Furthermore there is a new automotive safety assurance standard to comply with that reflects this change to a function-centric system view, ISO26262. Functional Safety is intrinsically end-to-end communication in scope. It has to treat the function of a subsystem as part of the function of the whole system. This means that whilst Functional Safety Standards focus on Electronic and Programmable Systems (E&PS), the end-to-end objectives for the approval process means that in practice functional safety review has to extend to the non-E&PS parts of the system that the E&PS actuates, controls or monitors.

Functional integration and this regulatory change are the issues driving a fundamental shift in how the HIL (Hardware-in-the-Loop) tool chain of automotive test departments has to be developed.

In the past, we would have to determine one HIL vendor before we set up a new HIL test bench to ensure that every particular subsystem can seamlessly work together with each other. Today we are moving from this all-in-one solution with monolithic HIL test benches provided by one single vendor towards heterogeneous and distributed test benches, which consist of several hardware modules from different HIL vendors, connected via the real-time capable HIL-Bus.

Why? Because no single HIL vendor has this previously mentioned all-in-one-solution, which meets all our test demands regarding distributed functions and highly integrated ECUs. As a result, we must choose the best-in-class solution for each sub-system and use those to develop a new test platform in which we have a high degree of confidence. The challenge is in how we go about integrating this set of HIL platforms from all of these different vendors in order to produce a new generation test bench for next generation cars and functions.

DDS-based HIL-Bus

Audi HIL test lab – showing how we integrate multi-vendor HIL systems together

The communication in cars has already moved from dedicated wire-based communication to a data-oriented bus communication using for example CAN bus or FlexRay. We have now transferred this bus-based approach from our cars to our next generation HIL architecture. We call this new approach ‘HIL-Bus based’.

Architectural view of the distributed HIL environment

Architectural view of the distributed HIL environment

To realize this bus-based approach for HIL-test-benches we need a data-centric bus representation mechanism to be the conduit of state information.

For the technical realization Audi decided to use RTI Connext DDS with integration points for HIL vendor systems.

RTI not only provided us a market leading implementation of DDS with their Connext DDS product, but their OCS (Open Community Source) license model gave us the ideal commercial framework to work within to develop an open market ecosystem for the HIL-Bus concept. OCS enables our HIL-Bus partners to have free access to RTI Connext DDS for their development and deployment. It thus removes a major inhibitor to adoption across the industry. It allows partners to focus resources on integration and quality.

Additionally we drive and focus on open international standards like the ASAM XIL-API to seamlessly integrate test automation software for 24/7 automated and deterministic tests and experimental software tools for manual testing.

Today we are working with multiple HIL system vendors to evolve this ecosystem and to instantiate the HIL-Bus as the ideal method for end-to-end functional system test.

For more information on HIL-Bus testing, we suggest this joint Audi/RTI article by Bettina Swynnerton of RTI and myself that was published in ATZ Elektronic in July 2014.

To learn more about ASAM XIL-API visit the ASAM website www.asam.net.

Building flexible manufacturing systems for Industrie 4.0 Reply

Many discussions on the industrial internet of things (IIOT) describe how all kind of sensors will be connected to the cloud, where the big data analytics beast will consume lots of data to provide you with efficiency optimizations. Huge cost savings are especially promised in the energy and transportation sectors. The medical industry, on the other hand, sees an opportunity to provide better and safer care, by integrating patient monitoring devices, and correlating the data or by merely reducing the amount of erroneous alarms.

I am very excited about how the IIOT will transform how things are made. Additive manufacturing and 3D printing are already revolutionizing how machines produce individual items. A generic 3D printer, supplied with the right basic materials, and guided by an electronic blueprint, can produce any type of component. NASA demonstrated recently how to create a new wrench in space, based upon a blueprint emailed to the international space station. GE is building critical jet engine parts using 3D printing. This technology is already here.

Hot off the 3-D printer is this jet engine bracket – GE Tumblr

The manufacturing floor will undergo dramatic changes. Manufacturing heavyweight Germany jumped on the opportunity and launched the Industry 4.0 project to lead in the fourth industrial revolution. In the US, the Smart Manufacturing Leadership Coalition initiative is bringing together manufacturing companies, and research groups to define the future of manufacturing.

Industrie-4.0-videos by Siemens and Bosch provide great examples of how changes in manufacturing will provide cost savings, as well as allow companies to deliver higher quality products:

  • More flexible manufacturing processes and shop floor systems will make it less expensive to deliver customer-defined products (“batch size 1”): custom LeBron James shoes, anyone?
  • Predictive maintenance will save companies money by extending the lifetime of the machine, and promises to reduce the overall machine downtime.
  • Manufacturing will be better integrated with spare part replenishment through location and weight sensors. This saves time and money to locate and reorder parts.
  • RFIDs and other technologies will provide a “product memory” to read what needs to be done to a part and record the progress: e.g., intelligent tightening tools record location and amount of torque applied to bolt on an airplane wing. By making the tools smarter, fewer mistakes will be made on the manufacturing floor. In the event of a product recall, it will also be easier to recall select products, rather than e.g., all cars of a specific model between 2007 and 2013.

Building a more flexible and integrated manufacturing system requires changes to the machines and addition of new sensors.  It  also requires changes to processes and software providing the Manufacturing Service Bus (MSB).

A recent whitepaper “Increasing the Adaptability of Manufacturing Systems by using Data-centric Communication” makes the case that data-centric communication paradigm is key to meeting the requirements of future adaptable manufacturing systems. We agree that a data-centric approach is key. However, obviously biased, we disagree with the authors on the solution.

RTI Connext, based upon the Data Distribution Service (DDS) OMG standard, meets the requirements to power next-generation manufacturing systems.  With RTI Connext, systems are:

    1. Decoupled – Designed for real-time communication of distributed systems, RTI Connext decouples systems and components in space, time and flow. Systems do not need to know in advance which component is producing the data or which component will be receiving the data. Historical data is available to late joiners. Sending and receiving data can be decoupled from the main control flow, avoiding blocking more critical tasks.
    2. Auto-Discovery – DDS discovers data sources and sinks through a lightweight automatic discovery mechanism. A keep-alive mechanism verifies that systems are still up and behaving as expected. Non-responsive components can be ignored.
    3. Resilient to failure – Because all information flows peer-to-peer, RTI Connext avoids any singlepoint of failure. Furthermore, RTI Connext is smart enough to handle redundant producers, consumers, and even networks.
    4. Proven in heterogeneous systems – RTI Connext is deployed in many many systems across many industries, it supports a wide variety of operating systems, including embedded real-time operating systems.  It also support many different transports; with its pluggable transport interface, new or legacy transports can be added.
    5. Standards-based – Because it is based on the DDS and Real-Time Publish-Subscribe interoperability protocol OMG standards, RTI Connext allows systems to interoperate in a standards-based way.
    6. SecureRTI Connext secure DDS and secure transports provide full security of all dataflows.  This includes authentication, encryption (confidentiality), integrity, and availability, along with non-repudiation of transactional information.
    7. Flexible – RTI Connext supports multiple communication paradigms: publish-subscribe (e.g., for alarms and events), request-reply (e.g., status inquiries), at-most-once, and exactly-once delivery. The quality of service can be defined on a per-topic basis: e.g., alarm data must be delivered reliably and high priority, while a sensor signal may only require best-efforts delivery at background priority.
    8. Fast and scalable – Real-time performance is in our veins: we design for low latency, low jitter, high throughput and high scalability. Our customers require micro- or milli-second latency, which scales from a handful of devices to hundreds and thousands.

Does RTI Connext provide all the pieces today? Not quite yet. Integration with legacy protocols and systems will be key. One option to bridge to the existing installed base is through the RTI Connext Routing Service, which provides a mechanism to build adapters and transformations between various existing protocols.

The changes happening on the manufacturing floor are exciting, and also demanding. A proven real-time communication protocol for distributed systems, like RTI Connext, will be key. Go check out why to use RTI Connext for your next generation manufacturing system.

From College Students to Entrepreneurs Reply

this post was written by the winners of the TechChallenge: Israel Blancas Álvarez, Ignacio Cara Martín, Nicolás Guerrero García, and Benito Palacios Sánchez.

A year ago, we started our work for the IV ETSIIT Technical Challenge (video). Who are we? Well, we are four students studying Electrical Engineering and Computer Science at the University of Granada in Spain.

figure1

Figure 1. Prometheus Team: Nicolás, Israel, Benito, and Ignacio.

Our team, Prometheus, won the Tech Challenge sponsored by RTI. For this challenge, teams of four or five students had to create a product to solve a challenge proposed by an external company. The theme for this year’s challenge was “Multi-Agent Video Distributed System.”

We joined this challenge for practical experience. One year after the Tech Challenge, we are all still students, but we  have now researched a business opportunity, designed the competitive product Locaviewer, developed a strategy to sell it in the market and created a working prototype in addition to the course work required for our degrees.

Locaviewer

Most parents with children in a nursery school worry about their children’s health and progress. Our product, Locaviewer, attempts to provide parents to track and see their child in real time. As part of our marketing plan, we created a promotional video. Our code has been released under MIT license on GitHub.

Team Organization and Schedule

The project took us approximately 250 hours to complete. Each week we met for at least 4 hours, with the exception of the last month where we spent 20 hours/week on the project. To be more efficient, we divided in two teams. Two people worked on the indoor bluetooth location algorithm. The other two focused on an application to capture, encode/decode a video stream, and share it using RTI Connext DDS.

Location Algorithm

The first and most important step of our solution was to determine the location of the children inside the nursery school. Each child needed to wear a wristband with a Bluetooth device -sensor-, which continually reported the signal power received a Bluetooth device -dongle-, placed in the room walls. This Received Signal Strength Indication (RSSI) value is usually measured in decibels (dB). We determined the relationship between RSSI and distance.

Figure 2. Empirically measuring Bluetooth signals by angle and distance.

Figure 2. Empirically measuring Bluetooth signals by angle and distance.

Figure 3. Locaviewer wearable.

Figure 3. Locaviewer wearable.

The RSSI values were transmitted to a minicomputer (Raspberry Pi or MK802 III) to run a triangulation algorithm and identify the child’s location. Since we knew the camera position, after determining the position of the child, we knew which cameras were recording the child and selected the best camera.

Figure 4. Indoor triangulation.

Figure 4. Indoor triangulation.

Video Recording Application

To record, encode, decode and visualize video we used GStreamer for Java. We tried other libraries such as vlcj but they didn’t support Raspberry Pi or satisfy the real-time constraints of our system. After some research, we discovered  GStreamer which worked with Raspberry Pi, and could easily get the encoded video buffer in real time (using AppSink and AppSource elements). This allowed us to encapsulate it and send it to a DDS topic. We worked on this for several months, even , implementing a temporary workaround with HTTP streaming using vlcj until we settled on our final approach.

We used the VP8 (WebM) video encoder. Since the wrapper for Java only works with GStreamer version 0.10, we could not optimize it, and had to reduce the video dimensions. Our tests used Raspberry Pi, but we plan to use a MK802 III device in the final implementation because it has the same price but more processing power. The final encoding configuration was:

Figure 5. GStreamer pipeline to record, encode and get video.

Figure 5. GStreamer pipeline to record, encode and get video.

We used the following Java code to create VP8 encoder elements.

    Element codec = ElementFactory.make("vp8enc", null);
    codec.set("threads", 5);
    codec.set("max-keyframe-distance", 20);
    codec.set("speed", 5);

    Element capsDst = ElementFactory.make("capsfilter", null);
    capsDst.setCaps(Caps.fromString("video/x-vp8 profile=(string)2"));

On the client side, we used the following configuration:

Figure 6. GStreamer pipeline to set, decode and play video.

Figure 6. GStreamer pipeline to set, decode and play video.

We used the following Java code to create VP8 decoder elements.

    String caps = "video/x-vp8, width=(int)320, height=(int)240, framerate=15/1";
    Element capsSrc = ElementFactory.make("capsfilter", null);
    capsSrc.setCaps(Caps.fromString(caps));

    Element queue = ElementFactory.make("queue2", null)
    Element codec = ElementFactory.make("vp8dec", null);
    Element convert = ElementFactory.make("ffmpegcolorspace", null);

We also tried JPEG encoding, but this was not feasible for real-time use due to the larger size and greater number of packets.

DDS Architecture

The publish-subscribe approach was  key to our solution. It allowed us to share data between many clients without worrying about network sockets or connections. We just needed to specify what kind of data to send and receive. We created a wrapper library, DDStheus, to abstract DDS usage in our system.

Figure 7. General DDS architecture of the system.

Figure 7. General DDS architecture of the system.

Our final solution was composed of six programs that shared three topics. We used different programming languages:

  1. Python to work at low level (HCI) with Bluetooth devices
  2. MATLAB/Octave to make the triangulation script
  3. Java to work with RTI Connext DDS and graphical user interfaces

We needed to know all the RSSI values in a room. We created a script to configure the Bluetooth dongles and get the RSSI information. These values were sent to a Java program using a simple socket connection in the same machine. The Java application published the data in the Sensor Data topic. It sent Child ID (the sensor Bluetooth MAC), Bluetooth dongle ID and position, current room (as a key to filter by room), RSSI value and expire time.

Figure 8. Sensors program flowchart.

Figure 8. Sensors program flowchart.

After  the cameras recorded and encoded the video, the Java program Gava sends the video viathe Video Data topic. It sent the camera ID as key value to filter the stream using ContentFilteredTopic with camera position, room, encoded frame and codec info.

Furthermore, the application put the camera ID, room and camera position in the USER_DATA QoS value of each video publisher. The triangulation minicomputer could then get all the camera info in a room just by discovering publishers. It could also detect new and broken cameras in real time and update the location script to improve the camera selection algorithm.

Figure 9. Video program flowchart.

Figure 9. Video program flowchart.

In last step we processed the data and wrote the result as Child Data topic. This was done by the room server (implemented with Raspberry Pi or MK802 III)  that triangulated the child location and selected the appropriate camera. It filtered only the sensors in the current room and gathered all video publisher info in that room. The data was sent to an Octave script, which returned the child’s location and best camera ID. The information sent to the cloud with the topic Child Data, included child ID, video quality, camera ID, child location and room ID. For efficiency, the child ID and quality are sent as keys that can be filtered against or used for sorting video.

To optimize the application, the room server called the triangulation script only if there was a subscriber asking for the child. We determined this using subscriber discovery and looking at the ContentFilteredTopic filter parameters.

Finally, we implemented a redundancy mechanism to handle the room server failure. Each minicomputer in the room created a publisher and set its USER_DATA value to the room and a default (unique) priority ID. If one of the minicomputers detected that it had the lowest ID in its room, it started the server application, and acted as server until a new minicomputer with a lower ID appeared.

Figure 10: Room server program flowchart.

Figure 10: Room server program flowchart.

User Applications

We developed two end-user applications. The first one will be used by the parents to see their children in the nursery school. The second program will be used by nursery employees to see all the cameras in real time, manage parent access (add and remove) and automatically handle attendance control.

Figure 11. Parent client application.

Figure 11. Parent client application.

Figure 12. Security camera program for the nursery.

Figure 12. Security camera program for the nursery.

Final Thoughts

We had to cope with two big problems in the challenge:

  1. Getting the RSSI values: we bought a very low quality, low cost Bluetooth device (around $5). The signal had a lot of errors and noise. We had to develop an algorithm to optimize the values, reducing that error from 3 to 0.5 meters. We could not find any library for low level operations with Bluetooth devices in Java (we finally used pybluez). We had to communicate using Python and Java programs.
  2. Video encoding: it was not easy to find a library that allowed us to get the encoded video buffer. It was even more difficult to optimize the elements in the GStreamer 0.10 pipeline to work at maximum performance in the Raspberry Pi. With the final configuration, the image delay is around 3-5 seconds. For better performance, we plan to replace the Raspberry Pi with a similarly priced MK802 III device, which includes Wi-Fi and a dual-core Cortex A9 processor.

RTI Connext DDS saved us a lot of work by implementing networking, data serialization and quality of service mechanisms. We thank our engineering school and RTI for giving us the opportunity and resources to successfully address this business challenge.

Connecting Your DDS Apps to Web Services using Javascript and Node.js Reply

One of the benefits of the Industrial Internet for manufacturers and OEM’s is the access to live data for more and more applications.  This data availability enables performance analytics, more robust business metrics, predictive maintenance analysis and various other capabilities to be realized by end users and equipment manufactures alike.  In the enterprise, Web Services dominate the typical access to data.  On the manufacturing floor or in deployed systems, real-time data delivery is a requirement.  DDS is one of the primary infrastructures used to share large amounts of data in very low latency delivery times.

Typical solutions for data analysis and visualization rely on protocol libraries and heavyweight client applications to receive / process the data.  Today, in the enterprise, the distribution of these proprietary client applications is cumbersome and time consuming.  This results in a reduction of the utilization an organization can get from available live data.  Thus, there is a need to enable applications to leverage technologies that can be found on everyone’s desk.  Just about every computer, tablet or handheld available today have browsers like Internet Explorer, Firefox, Safari, Chrome and others.  Through the use of standard web technologies like Javascript, JQuery, Node.js and Websockets, it is quite trivial to interface browser based user applications with live data.  The problem, however, is how to connect live DDS data up inside these thin client browser based applications.  As we know, access to DDS data requires the use of DDS libraries to interact with the DDS RTPS protocol.

To solve this problem, we have created a lightweight connector technology that enables DDS data to be accessed with Javascript, Node.js server side applications.  This solution provides the ability to have a single server process interact bidirectionally  with DDS data while serving up html pages that leverage standard WebSockets for communication to the server for data.  Take a look at the following diagram showing how the Javascript connector enables the flow of data between DDS and browser based applications.

Image1

 

Taking this a step further, and now that there is an integration with Node.js, we have the ability to create useable object nodes in NodeRed.  NodeRed is a visual programming environment for Node.js applications.  This development environment, which itself runs on Node.js, can be used in any browser.  NodeRed provides many building blocks for typical items used in Industrial Internet applications such as:  WebSockets, HTTP, custom TCP, custom UDP, MQTT, custom function blocks, email, twitter, XMPP, and many others.  As you can see in the following diagram we are using several of these blocks to subscribe to DDS data (Square, ShapeType) and publish data on MQTT, then with a separate flow, subscribe to the MQTT server and output the data to a debug window.

Image2

At RTI, we are constantly working on creating components that are useful and beneficial for your Industrial Internet applications.  This latest solution gives a developer quick and easy access to live data in many different forms depending on their specific application requirements.  The ability to present live data in a thin client browser based application provides a great deal of flexibility for creating and distributing applications to better leverage data that is already available in the system.

Information regarding NodeRed can be found at http://www.nodered.org.  If you are are interested in connecting up your live data to backend enterprise applications and thin client browser based applications, please contact us via email or leave a comment in the comments section.

A Noteworthy and Newsworthy 2014 Reply

2014 was an especially busy year here at RTI. We continued our dedication to innovation in the Industrial Internet of Things (IIoT), celebrated wins with some great new customers, hosted events all over the world, and more. A lot more, really!

To cap off the year I wanted to share with you some of the great things that happened over the past 12 months and so – drumroll, please! – here are some of the most noteworthy and newsworthy things that happened this year at RTI:

London Connext Conferencelondon connext conference rti EMEA 2014

This October at the packed London Connext Conference, I heard more than 20 customers share how they used our products in building solutions to their hard problems. Coding tips, illustrative guidance, professional peer conversations and live demos during the event cemented relationships between established and budding DDS experts. It was inspiring and exciting, and I can’t wait for next year’s conference!

To see what all of the excitement was about, head on over to community.rti.com for an agenda summary and selected presentations.

RTI In the News

IoT-Influence_10_2Forbes Reports RTI as the #1 Influencer for The Industrial Internet of Things

For years, we’ve worked with our customers and partners to develop leading solutions for massive and complex problems in the healthcare, energy and technology markets. In recent months, we’ve seen big rewards for our hard work.

In July 2014, Forbes published a study conducted by The Data Journalism Group at Appinions that identified the top influential companies for the Industrial Internet of Things. RTI was rated number 1. Check out the Forbes article and the Appinions IoT Influence Study (Industrial IoT rankings on page 9).

Obviously, we’re  thrilled to be recognized alongside companies such as GE, Cisco, Google and Apple for our leadership and progress in the market.

CIOReviewCIO Review Names RTI as One of Top 50 IoT Companies

In December, CIO Review named RTI a most promising Internet of Things company, emphasizing the performance, reliability and security of Connext DDS as a real-time communications platform for smart integrated systems. (link to pdf – http://www.rti.com/docs/CIOReview-2014.pdf )

The Industrial Internet ConsortiumIndustrial-Internet-Consortium-Member-logo210514

In April, RTI joined the Industrial Internet Consortium (IIC), which has since grown to100+ industry-leading companies. Its goal is to accelerate the development of connected industrial applications and to provide a common blueprint for the IIC.

RTI CEO Stan Schneider was elected to the IIC steering committee for a 1-year term as the representative for small industry. The IIC offers real market opportunities for motivated and innovative smaller businesses to work closely with leading companies and government and advance the future of industry.

Working With Our Customers (who are doing amazing things!)

In 2014 we announced several important customers:

  • GE Healthcare is using the RTI Connext DDS platform in smart systems of connected medical devices in hospitals and to enable faster integration of large medical instruments.
  • LocalGrid Technologies is using Connext DDS and Connext DDS Secure to deliver high performance scalability to the electric grid. LocalGrid is modernizing the electric grid to support distributed generation of renewable energy. Bob Leigh, CEO of LocalGrid, says “The LocalGrid DataFabric™ grid architecture enables [distributed generation] by securely supporting field deployment of analytics and control applications.”
  • The U.S. Army and General Dynamics Advanced Information Systems selected RTI technology to be integrated into a prototype next generation open architecture (OA) for Unmanned Aircraft Systems (UAS) Ground Control Stations.
  • The European Space Agency built an advanced telerobotics development platform on RTI Connext DDS. We are thankful to work with companies who are leaders in their markets and we’re excited for the projects we have in progress.
  • The MEVION S250, the world’s most advanced compact proton beam radiotherapy system, was recently used – for the first time – to treat a cancer patient. It uses Connext DDS for data distribution and control systems. Get the details

 Our Top SlideShare Content

Top Content can mean many things – most clicks, views, likes, shares, etc. These 2 SlideShares rocked in 2014 and we know this because you – our users – told us (and the numbers back you up)!

Understanding the Internet of Things Protocols

Comparison of MQTT and DDS as M2M Protocols for the Internet of Things

New Releasesconnext dds 5.1.0_wordCloud

2014 saw the release of Connext DDS 5.1 and Connext DDS Secure, two major launches with usability enhancements, new capabilities and an eye towards future customer development needs.

  • Connext DDS Secure. The OMG DDS Security standard was approved in April and the new RTI Connext DDS Secure package – the world’s first complete DDS security solution – is available as a preview release. Read more at www.rti.com/gosecure [link may change] Read more about Connext DDS Secure
  • Connext DDS 5.1. Focusing on greatly increased usability, scalability and performance, Connext DDS 5.1 has been available since February 11. Improvements abound, with over 60 new features and enhancements, as well as support for over 20 new platforms. Read more and download Connext DDS 5.1
  • VS2013 Connext DDS Downloads. Current customers can download Connext DDS Professional libraries for Microsoft Visual Studio 2013 on both 32-bit and 64-bit Intel architectures. Get the VS2013 files now
  • Performance Test Update. Connext DDS 5.1 contains powerful new auto-tuning capabilities that dynamically balance latency and throughput as system conditions such as update rates and network capacity change. With the ” -enableAutoThrottle” and “-enableTurboMode” command line options, you can see how these new features perform on your hardware and network. Try out Performance Test now
  • DDS Toolkit for Labview. Our DDS Toolkit for LabVIEW is completely free from the NI LabVIEW Tools Network. Sara Granados, an RTI software engineer, explains just how easy it is to use, and how valuable it can be.  Watch the DDS Toolkit for LabVIEW video
  • Connext DDS Micro and Connext DDS Cert.

The Blog

In our top blog post of 2014, Alex Campos, Senior Software Engineer at RTI, explained how to Create a P2P Application with Fewer than 35 Lines of C++11 Code. Other popular posts included:

Looking Ahead

This year has been chock full of product advancements, leading customer announcements and top accolades. We are well-positioned for 2015 as we ramp up our efforts with our employees, partners, the IIC, the OMG and our customers. Together we’re all creating a more intelligent IIoT that promises to continue to positively impact the world.

Will Work for User Feedback! Reply

We just attended a Connext DDS Users Group meeting in Chengdu, China on November 14. Most of the conference was conducted in Mandarin Chinese, and neither Edwin (RTI VP of Sales) nor I spoke the language.

However Vision Microsystems, our host and RTI’s Chinese distributor, had arranged an excellent translator who managed to help us keep up with about 75% of the information. This was their second such meeting, with 70 users representing over 30 Connext DDS projects.

Howard Wang

Taking the stage this November in China at our latest DDS Users Group event.

Edwin and I took part of the day to explain how RTI was involved with the Industrial Internet of Things, and presented a technical roadmap of the Connext 5.2 release coming in early 2015.

During the meeting, several long-time Connext DDS developers explained how they chose DDS for their projects, the problems they were solving, and pointed out issues they had with DDS. These presenters were pragmatic and direct, pointing out situations where DDS was not the best solution for them. Both Edwin and I were glad to hear these points of view. Neither of us, nor RTI, advocate DDS as the best technology for all applications involving network communications and systems integration.

7DyPJ

Edwin, our VP of Sales, engaging the attendees.

We learned that many use cases incorporating Connext DDS were those in which DDS was superior to legacy or competing solutions. Users were discovering better performance and scalability, and reducing user code for implementing many capabilities and features — particularly those that were inherently part of our stable, thoroughly-tested, commercial-off-the-shelf technology.

Overall, the testimonials were gratifying for us, and valuable for others in the audience.

This type of feedback helps assure our potential customers that Connext DDS solves real problems in real systems. RTI generally focuses on problems that are difficult to solve or would otherwise take a great deal of effort to solve with DIY solutions. For Edwin and I, the Chengdu meeting was a chance to understand how users truly employed our products and validate their experiences as their discovered where DDS wasn’t a perfect fit.

Of course, in the future, and with more customer feedback, DDS is quite capable of becoming a better fit for many more problems than it solves today. You can help make that happen by sharing your experiences with Connext DDS. Post them to our community forums, or send an email to info @ rti.com .