JSON and Binary Topics for Real-Time Messaging

In Diffusion^TM 5.7 some new topic types have appeared, namely JSON and Binary. More generally these are referred to as the first examples of universal topics. In this article I will describe what these universal topics are and why they represent a huge step forward for Diffusion users. Everything discussed here also applies to Diffusion Cloud.

Prior to 5.7, Diffusion had a number of different topic types, representing different data formats. Each type has a different way of configuring, creating and using the topics. Universal topics introduce a consistent and simple approach to topic creation and handling.

In addition, a great strength of Diffusion has been its handling of delta messages, specifically in its Record topic type. This means that when the state of a topic changes, only the difference between the old state and the new state is sent to the client, significantly reducing the amount of data that needs to be sent. The downside of using these deltas has been that the client application has to differentiate between delta messages and full values (sometimes known as topic loads or snapshots), and apply deltas to local topic representations. With universal topics there is no need for the client to know anything about deltas as even though they are used for transport, they are automatically applied to locally cached topic values and only values are streamed to the client.

Universal topics themselves are backed by different data type libraries. At the first release there are the JSON and Binary types, but more are on the way in future releases.

At the first release, data type libraries for JSON (JavaScript Object Notation) and Binary are provided. JSON was chosen as one of the first implementations because it is so widely used and understood in web applications. It also provides a very flexible and powerful data structure that supports maps and arrays as well as basic types. Binary support allows for the processing of raw unstructured binary content. Both benefit from the advantages of only transmitting deltas of change without the client application having to perform any special processing.

Generally, the application programmer does not need to know much about the data type libraries to use universal topics, but they are discussed in a little more detail below.

Examples are shown using the Java API but other APIs will have equivalent features. In the JavaScript^® API the handling of JSON content is of course even simpler.

Why Universal?

Two definitions of the word ‘universal’ are “applicable everywhere, or in all cases; general” and “used or understood by all” and these definitions sum up what the goal of introducing universal topics was.

The aim is that there will really be only one type of topic, but a topic will be backed by a specific data type library via the new DataType API.

In addition, the name is used to differentiate the new types and the way in which they are handled from the various original types.

Data Types

Backing universal topics are data type libraries. The DataType API allows the handling of data formats to be separated from topics such that the topic handling API is the same for all topics and the topic type simply identifies the data type that it uses.

So what is significant about a data type? Well, every data type has an internal format in which a value can be held, a value being what would be held as the state of a topic. The internal format is largely irrelevant to the user and can generally just be thought of as an array of bytes. In any case, the data type implementation will provide mechanisms for reading a value from an array of bytes or writing it out as an array of bytes. For JSON topics the value is represented by a JSON object and for Binary topics by a Binary object.

Data types also (optionally) provide support for delta processing via delta types. Each data type can have one or more delta types. What the delta type provides is the ability to compare (or diff) two values to produce a delta representing the differences between the values. It also provides the ability to apply a delta to a value to produce a new value.

Most data types will support binary delta (BinaryDelta) processing which compares the binary representation of two values to produce a binary delta describing the differences. There may also be support for structural deltas which take advantage of the internal structure of the data.

In Java, the easiest way to obtain a DataType instance is using convenience methods as follows:

JSONDataType jsonDataType = Diffusion.dataTypes().json();
BinaryDataType binaryDataType = Diffusion.dataTypes().binary();

The main thing about the DataType API is it provides universal topics with a standard (universal) interface for handling data that allows them to present a standard way to update and consume streamed data from these topics.

Value Streams

Diffusion provides client APIs that allow client applications to receive streamed data from topics as those topics are updated. These streams were known as topic streams (or in the classic APIs as topic listeners). When the topics were single value types (e.g. Strings), then these would simply present a stream of values to the client. However for more complex types that support deltas (e.g. Record topics) then the stream would present an initial value (snapshot or topic load) and then subsequently would present deltas describing only the changes from the last state. The responsibility for differentiating between values and deltas and applying deltas to local value representation was the responsibility of the client application. This could get complicated.

Now with universal topics there are value streams. The most significant thing about these streams is that even though deltas may still be getting sent to the client from the server, the client only ever receives a stream of values. This is because a local cache holds the last value for each subscribed topic and as deltas arrive they are automatically applied to the last value so that an updated value can be presented. The second important thing about value streams is that they are typed which means that instead of simply receiving generic content and having to decode it yourself, the stream now presents an object of the type it was created for. Updates are filtered such that a typed stream will only receive value updates for topics that match its type.

So you can define a JSON value stream and register it for a selection of topics and that stream will receive all updates for JSON topics only as JSON values. No decoding and no applying of deltas required.

To create a JSON value stream in Java:

Topics topics = session.feature(Topics.class);
topics.addStream("?FX/", JSON.class, new JSONStream());

Values for all subscribed JSON topics under the FX root topic would be routed to the stream.

The implementation of JSONStream might look like this:

private class JSONStream extends Topics.ValueStream.Default<JSON> {
    public void onValue(
        String topicPath,
        TopicSpecification specification,
        JSON oldValue,
        JSON newValue) {
        LOG.info("{}={}", topicPath, newValue.toJsonString());
    }
}

The implementation above only shows the onValue method but there are onSubscription and onUnsubscription callbacks as with topic streams. One difference is that all callbacks will provide the TopicSpecification which includes the topic type and at future releases will also include any basic properties set for the topic.

You can add a value stream at any time and if any topics that the client is currently subscribed to match the stream then the stream will be notified of the subscription and the current value.

Even though value streams are essentially for use with universal topics they can also be used for single value topics which do not require delta processing. You can simply add a value stream with type Content and values for all universal topics and single value topics will be routed to it.

Value-Based Updating

Diffusion also supports the updating of topics with deltas at the server. This reduces the size of updates that are sent to the server and also avoids the need to diff a new value with an old value at the server to calculate the deltas to send to clients. The problem with this is that it does require that the updater of the topic has to encode the delta in the manner the topic expects.

Now, with universal topics there are value updaters which provide clients with the ability to update topics by simply presenting a new value for the topic at the client end. The value updater uses a local cache holding the last value sent for each topic so that when a new value is presented then it is compared to the previous value in order to generate a delta to send to the server.

A value updater may be obtained from the Updater that is provided on registration as an update source or in the case of non-exclusive updating directly from the TopicUpdateControl feature as follows:

TopicUpdateControl updateControl =
    session.feature(TopicUpdateControl.class);

ValueUpdater<JSON> updater = 
    updateControl.updater().valueUpdater(JSON.class);

And having obtained a value updater then updating is simply as follows:

JSON jsonValue = jsonDataType.fromJsonString(jsonIn);
updater.update("MyTopic", jsonValue, updateCallback);

Not only does this make updating easier, but also significantly reduces the amount of data that needs to be sent from the updating client to the server.

Creating Universal Topics

Previous topic types often had complex configuration and in some cases (e.g. Record topics) involved the definition of detailed metadata describing the data structure. This was done using a mechanism called topic details and details describing the topic had to be constructed before creating the topic. In the case of Record topics this could lead to a lot of code just to describe the topic layout.

Now, with universal topics, it is very simple to create a topic using the TopicControl feature. The simplest mechanism is just to provide an object defining the topic’s initial value:

TopicControl topicControl = session.feature(TopicControl.class);
JSON jsonValue = jsonDataType.fromJsonString(jsonString);
topicControl.addTopicFromValue("MyJSONTopic", jsonValue, addCallback);

Alternatively, a universal topic can be created without an initial value. Clients can still subscribe to it but would receive no value until the first update was performed.

For example:

topicControl.addTopic("MyJSONTopic", TopicType.JSON, addCallback);

There are very few additional options available with universal topics (only two at present) but if you do want to specify them then you can create a universal topic using a TopicSpecification that allows topic properties to be specified. So the most involved example of creating a universal topic would be something like the following:

TopicSpecification topicSpec =
    topicControl.newSpecification(TopicType.JSON)
    .withProperty(TopicSpecification.PUBLISH_VALUES_ONLY, "true");
topicControl.addTopic("MyJSONTopic", topicSpec, addCallback);

The specified property instructs the topic to only publish values and not take advantage of delta processing.

JSON Topics

The significant addition to the Diffusion toolkit that comes with universal topics is JSON topics. The JSON topic type (TopicType.JSON) is supported by the JSON data type (JSONDataType) and provides binary delta handling and some support for structural deltas.

The real power of JSON topics is the flexibility of the data structure. The topic value can be any valid JSON, therefore as well as basic values the topic value can contain maps and/or arrays of data values.

And since JSON is a subset of JavaScript it can be used directly in JavaScript client applications.

In Java the JSON data type implementation may be obtained as follows:

JSONDataType jsonDataType = Diffusion.dataTypes().json();

A JSON value can easily be created from a JSON string:

JSON jsonValue = jsonDataType.fromJsonString(jsonString)

And a JSON value can be converted to a string:

String jsonString = jsonValue.toJsonString();

Java applications can use a JSON library such as Jackson to create and interpret JSON represented as strings.

The internal value of a JSON object is not held as a JSON string but in CBOR (Concise Binary Object Representation) format to reduce memory and network overheads. It is possible, and usually preferable for applications to work directly with the underlying CBOR representation and thus avoid creating intermediate JSON strings. We recommend using Jackson’s jackson-dataformat-cbor library for generating and parsing CBOR format data or a binding library like jackson-databind to map CBOR data to and from instances of Java classes. The JSON.asInputStream method can be used to read the value directly into a CBOR parser. This more advanced usage will be the subject of a future blog.

JSON Structural Deltas

The JSON data type library also provides a structural delta type (JSONDelta) for use with JSON values.  These are useful for identifying small changes to complex JSON values.

A JSONDelta describes the differences between two JSON values. Unlike a binary delta, a structural delta can be queried to determine its effect. There are removed and inserted methods which provide full details of the differences between the two values. A JSONDelta instance can be created using the JSON.diff(JSON) method.

The following example shows how to use such a delta to filter changes in a value stream:

private class PriceStream extends Topics.ValueStream.Default<JSON> {

    public void onValue(
        String topicPath, 
        JSON oldValue,
        JSON newValue) {

        JSONDelta delta = newValue.diff(oldValue);
 
       if (!delta.removed().intersection("/price").isEmpty() ||
           !delta.inserted().intersection("/price").isEmpty()) {
 
           // The 'price' field has changed.
           processNewPrice(newValue);
       }   
    }
}

The key used to refer to items within the value is specified as a JSON Pointer which is a standard mechanism for referencing JSON data items.

Binary Topics

Binary topics can be used for arbitrary binary data where the responsibility for formatting and interpreting the data belongs solely to the application. JSON or single value topics should be used in preference to Binary topics as the application interface would be simpler. However they can be very useful for other unsupported types of data, such as images.

Binary topics support binary deltas so that only the changes between two values get transmitted. This can mean a significant saving in network costs for the transmission of some types of data.

What Next?

Universal topics are fully supported in Java, Android™ and JavaScript in Diffusion 5.7 and will be supported in all Unified clients soon.

In addition, more universal topic types will be added, starting with replacements for current topic types, such that existing applications can migrate to using universal topics with minimal effort. As new replacement types are added the old types will be deprecated.

Structural delta processing will also be expanded upon. This will include updating using structural updates and more availability of structural information at the client.

Universal topics also provide the basis for many new and exciting improvements to Diffusion. One of the first of these will be the enabling of fully automated conflation so that users will be able to benefit from conflation without writing any code.

Much more will follow, keeping Diffusion ahead of its competitors in realtime data delivery. If you would like to read more about our newest feature, JSON, and see how you could save data, check out our live demo here or learn more about Diffusion.