New Topic View Features in 6.4

Diffusion 6.3 introduced topic views, a mechanism to produce virtual topics, that refer to other topics in the topic tree for their value.

Topic views are an evolution and generalization of slave topics. A slave topic is derived from a single source topic in the topic tree. Sessions may subscribe to the slave without having any knowledge of the source topic. This allows additional topic tree structures to be set up without changing the base data model. However, slave topics can be cumbersome – to duplicate a segment of the topic tree then you must set up individual slave topics for each source topic.

Topic views introduce a new approach to the problem, mapping a selection of source topics (using Diffusion’s topic selectors) to ‘reference topics’ in another part of the topic tree. To the end user a reference topic is like any other topic (though they cannot be directly created, removed or updated), but its value derives from a source topic.

Recap of Topic Views in Diffusion 6.3

In Diffusion 6.3, reference topics paths can derive from the source topic path, the source topic’s value or from a constant value. For example, a simple topic view that maps one branch of the topic tree to another is:-

map ?a/ to b/<path(1)>

This specification maps all source topics under the topic node a to matching topics under the node b. For example, a/x/y/z will be mapped to b/x/y/z and so on. The directive path(1) means can be read “all path elements from index 1 and beyond”.

When using JSON topics it is possible to extract value from the source topic and embed it in the path of the reference topic. It is also possible to map a subset of the source topic value to the value of the reference topic. The following example shows both of these features in use:-

map ?accounts/ to balances/<scalar(/account)> as <value(/balance)>

In this case a source topic with a path of accounts/account1234 with a value as follows:-

{    
   "account" : "1234",    
   "balance" : { "amount" : 12.57, "currency" : "USD" }  
}

will be mapped to a reference topic with a path of balances/1234 and value as follows:-

{    
   "amount" : 12.57,    
   "currency" : "USD"  
}

Topic views can also map topic specifications and specifying throttling of reference topics but this is all covered in a separate blog – or see the product documentation for full details of how to specify topic views.

New: Topic View Expansion

Diffusion 6.3 provides a one-to-one mapping of source topics to reference topics. Every source topic selected by the view maps to exactly one reference topic.

Using the new expand directive Diffusion 6.4 can expand a JSON array or JSON object within the value of a JSON source topic to multiple, separate reference topics.

The basic syntax of the expand directive is

<expand(sourcePointer, pathPointer)>

The optional sourcePointer parameter is a JSON pointer to an array or object. If it is is not specified then the default is the root of the JSON value.

The optional pathPointer parameter is a JSON pointer indicating the reference topic path element. If it is not specified then the index (in the case of an array element) or the key (in the case of an object) is used instead.

These concepts will become clearer by means of examples:-

Expanding a Single Array

Let’s assume we have a JSON topic called allCars containing an array of all known cars in the following format

{
    "cars": [
      { "reg":"HY58XPA", "type":"Ford", "model":"Sierra" },
      { "reg":"PY59GCA", "type":"Fiat", "model":"Panda"},
      { "reg":"VA63ABC", "type":"Ford", "model":"Ka"}
    ]
}

Then using the following topic view specification:

map allCars to cars/<expand(/cars,/reg)>

creates reference topics with these paths:

cars/HY58XPA
cars/PY59GCA
cars/VA63ABC

The source pointer determines the value of the reference topic. For example, the value for reference topic cars/HY58XPA is:

{
   "reg":"HY58XPA",
   "type":"Ford",
   "model":"Sierra"
}

The second parameter is optional, for example:

map allCars to cars/<expand(/cars)>

In this case, the topic path is taken from the index of the current array element, giving us these topics paths:

cars/0
cars/1
cars/2

Use this topic view to group the cars by type:

map allCars to cars/<expand(/cars,/type)>/<scalar(/reg)>

In this case, the mapping will result in topics named as follows:-

cars/Ford/HY58XPA
cars/Fiat/PY59GCA
cars/Ford/VA63ABC

Note how the scalar directive is relative to the root of the expanded element, not the source topic value.

Expanding Objects

The expand directive can be applied to objects as well as arrays. For example, if topic people/jsmith has the value:

{
   "name" : "John Smith",
   "car" {
      "reg":"HY58XPA",
      "type":"Ford",
      "model":"Sierra"
   }
}

the following topic view specification can be used:

map ?people/ to <scalar("/name")>/car/<expand(/car)>

This creates source topic with values as shown:

We can use the expand directive many times to unpack complex source values that contain nested arrays or objects. Each expand directive focusses the evaluation context to successively smaller parts of the source value, so the JSON pointers in each directive are evaluated relative to the value produced by the previous directive.

Suppose the previous array example is extended so that each car can have multiple drivers, giving a possible values as follows:-

{
     "cars": [
       { "reg":     "HY58XPA", 
         "drivers": [{"name" : "Bill"}, {"name" : "Fred"}]
       },
       { "reg":     "PY59GCA",
         "drivers": [{"name" : "Jane"}, {"name" : "Fred"}]
       },
       { "reg":     "VA63ABC", 
         "drivers": [{"name" : "Tom"}, {"name" : "John"}]
       }
     ]
 }

We can expand both levels of the array hierarchy like this:

map allCars to cars/<expand(/cars,/reg)>/drivers/<expand(/drivers,/name)>

In this case the following reference topics would be generated with values as shown:

Caveats

Topic view expansion is a powerful feature which should be used with care to avoid performance issues.

When expanding a JSON topic containing an array or an object, there should be a strong correspondence between successive source topic values. There are two considerations here: the set of topic paths derived from the expanded topic value should not be too volatile, and each pair of elements in the old and new structures should be related.

A stable set of topic paths is important because each time an element is added to a source topics value, a reference topic is created; and every time an element is removed, a reference topic is removed. Rapid changes to the derived topic paths leads to a high rate of topic additions and removals, which has a cost. Changes to the topic tree are processed asynchronously in the background thread pool, and can therefore be a source of contention. Additionally, short-lived topics are less useful as an application data model and can be confusing to subscribers.

For each pair of corresponding structure elements in the old and the new source topic values, the server calculates a delta – just as it does for an update of a topic value. The computational cost of the delta calculation is proportional to number of bytes that differ in the old and new values. This cost is rarely a problem for simple topic updates, but topic view expansion could potentially trigger a much larger number of comparisons for a single change to a source topic value. If there are many differences between the structure elements in the old and new source topic values, the input thread processing the update can become a bottleneck. On the other hand, topic view evaluation will be efficient if only a few of the elements have changed, or if the changes between the elements are minor.

Infrequent major change between successive source values can be accommodated, perhaps with an observable increase in latency before all of the reference topics reflect the change. However, if each value is very different from the previous value, the server may not be able to keep up with the rate of source value updates.

Summary

The new expand directive can be used in many different ways to map arrays or object values to multiple reference topics, bringing new powerful functionality to topic views.