Persisting data in a Real-Time distributed system

Jens Pillgram-Larsen — Fri, 26 Jun 2009 17:29:45 +0000

Most of the distributed systems we deal with at RTI have performance constraints at their core. Either the system is pushing the limits of the available resources, or the action-reaction timing is critical for a given event. In other words the constraint might be on throughput or latency (or increasingly latency vs. throughput). In these kinds of systems persisting data is a real challenge. In many systems it is becoming a requirement that the distributed data is persisted. Take for example flight systems and automated trading systems, where persisting data is necessary to adhere to regulatory demands. At RTI we have set out to make persisting distributed data as minimally intrusive, performant and configurable as possible.

Recording Data

The simplest case of persisting data in a distributed system is recording the data to a file system. This can be done locally for each node, or centrally on a “recording” node. Each system setup solves different problems. Persisting local state on each node allows you to recover from a local crash by reading that state during recovery. Recording all the data on a centralized node allows you to query the entire set of data recorded (not just the subset any node would see) to find patterns or anomalies, or do a post-mortem on a system failure.

Locally each RTI Data Distribution Service publisher or subscriber keeps samples in queues. To allow for local persistence a mirror of these in-memory queues can be kept on disk. After a crash the publishers and subscribers can then be created with the same state as they had before the crash.

Centrally you can run RTI Recorder that can be configured to subscribe to interesting data (or all data for that matter) and write it to a file system. This data can then be queried either in real-time or during off-peak hours. As an aside RTI Data Distribution Service and RTI Recorder have countless flight hours on mission critical systems!

Connecting the Distributed System to an Enterprise Database

In some cases simply having the data on a file system is not enough — the full power of an enterprise database is needed. At RTI we have solved this by having RTI Real-Time Connect for ODBC enabled databases. Now you can use RTI Data Distribution Service to implement a distributed database, or utilize a database for your persistent storage needs. This allows you to use any tool or query you are familiar with to work with a distributed set of data. The possibilities are literally endless!

No matter what the use-case — recovery, logging, post-mortem analysis or database integration — persisting real-time distributed data is here to stay. The challenge is persisting the data with minimal intrusiveness on the real-time aspects of the system. At RTI we work hard to make that happen.

Complex Event Processing – Making sense of all your data

Jens Pillgram-Larsen — Thu, 28 May 2009 00:41:52 +0000

So you have a distributed system and you’re happily sending data between nodes in your system. The consumer applications are consuming the data your producer applications are producing, and everything is running smoothly. Now, that doesn’t sound like any system you know does it? Distributed systems are by nature complex. Nodes and applications are not straight producers or consumers; they’re a bit of both. And there’s always some resource contention. This is where Complex Event Processing (CEP) comes on the scene. CEP allows you to run queries on streams of data in real-time, either transforming the data or triggering alerts based on data content. Let me explain by talking about a couple of use cases.

Let’s say you have a large distributed system where data is being exchanged between nodes and applications. You’re finding that some of your applications are getting starved for data, but everything looks fine in the network – there are only a few lost packets and plenty of bandwidth available. The next step is moving up the stack to see what’s going on with the networking middleware. Now, if you’re lucky enough to have a well-behaved middleware like RTI Data Distribution Service, there are ways to retrieve detailed statistics about the communication channels. Passing this information through a CEP engine you can write queries that determine when and where you have a slow producer or consumer in your system that is clogging everything up.

Another common use-case is event correlation. Seemingly disparate events may not be, and corroborating can trigger pattern based alerts. Anyone who’s watched a Hollywood spy movie knows this – increased cell-phone chatter coupled with certain keywords and the change in behavior of suspected operatives means something bad is likely going to happen. Using CEP events can be monitored and corroborated in real-time and alerts sent out when the situation warrants it.

CEP is an incredibly powerful technology, and with increasing data rates it is a technology that should be watched carefully in the next couple of years.

Go check out RTI Event Processing at http://www.rti.com/products/rtievent_processing/ for an integration between RTI Data Distribution Service and Complex Event Processing.

Blogs from RTI » Jens Pillgram-Larsen

Persisting data in a Real-Time distributed system

Complex Event Processing – Making sense of all your data