Topic Views in Release 6.8 – Conditionals and Calculations

Release 6.8 of the Diffusion Intelligent Data Platform for real-time applications includes new additions to the popular data-wrangling feature called ‘topic views’. It is now possible to include conditional processing of JSON topic values, and also to perform calculations on numbers within the JSON and inject the results into the output values.

What’s new in topic views?

A Diffusion user can define ‘topic views’ which are able to dynamically manipulate the topic tree to generate new topics derived from a selection of input topics. Topic views are able to map to new topics, whilst optionally transforming the data in those topics, as well as providing a number of other capabilities such as throttling or delaying publication of updates. Topic views are specified using a proprietary DSL (Domain Specific Language) which allows complex topic tree manipulation to be performed without writing any code.

A previous blog for the 6.7 release included a brief history of topic views, summarising the various data-wrangling features available. Release 6.7 introduced new patch and type clauses and now in release 6.8 the functionality has been extended to provide a new process transformation clause that allows conditional processing along with calculations.

There were previously two types of transformation clauses that could be specified in a topic view after the path mapping. These were insert and patch. A new transformation called process has now been added which supports performing some processing on the input JSON value (which may be conditional) to produce a modified out value for the generated reference topic.

The new process transformation

An example of a process transformation in a topic view specification is shown below :

map ?a// to b/<path(1)> process {if "/Price lt 50" set(/Tier, 1) elseif "/Price gt 50" set(/Tier, 2)};

The format of the process transformation clause itself is:-

process {statement}

Where the statement can be one or more operations, or a condition statement comprising one or more conditions with operations to perform if they are satisfied.

Operations

The following operations are currently supported :

• set(pointer, value)
  Sets the field indicated by the JSON pointer to an integer, string, or boolean value.

• set(pointer, calc "calculation")
  Sets the field indicated by the JSON pointer to a value which is the result of the specified calculation.

• remove(pointer)
  Removes the JSON item at the specified pointer.

• continue
  A special operation that indicates that the topic view evaluation should continue with the value as it is. Used with conditionals.

Operations can be chained by separating them with a ‘;‘ as shown in the example below:-

set(/Amount, calc "/Value * /Number"); remove(/Value); remove (/Number)

Operations are performed in order on the original input value and if any operation fails a reference topic would not be created. However, a remove operation does not fail if the referenced item does not exist.

Calculations

A calculation may be specified as the value of a set operation. A calculation is a simple arithmetic calculation upon integer fields. If applied to a non-integer field the evaluation will not proceed.

Arithmetic operators supported are +, - , * and /.

Examples of calculations are:-

set(/Value, calc "/Value * 2")
set(/Result, calc "/Value / 2")
set(/Bonus, calc "/Salary + 1000")
set(/Bonus, calc "/Salary + 1000 + /Age * 10")

Brackets may be used to override operator precedence.

Conditional statements

A conditional statement is made up of an if clause, optionally followed by one or more elseif clauses and an optional final else clause.

The if clause takes the form :

if "condition" operation(s)

Conditions are described in more detail below. If the condition is satisfied, the operations are applied to the value and the process is complete. If the condition is not satisfied, processing moves on to any elseif or else clauses that follow, but if there are none, the topic view evaluation does not proceed and no reference topic is created.

The elseif clause takes exactly the same form as the if clause. The else clause is simply followed by operations to be performed if none of the previous conditions were satisfied. The continue operation can be used with the else clause to indicate that the input value should be carried forwards as it is.

Conditions

A condition is of the form:

pointer operator [constant/pointer]

Where the operator is a relational operator and constant is a string, integer, or boolean value. for example:-

/Age > 40
/Name = "Bill"
/Age > /RetirementAge
/Manager eq true

Supported operators are = (or eq), > (or gt), < (or lt),  != (or ne), >= (or ge), and <= (or le).

Compound conditions are supported by means of boolean operators | (or or) and & (or and).

For example :

/Age = 50 or /Age > 80
/Age gt 50 & /Department eq "Accounts"

Normal boolean precedence applies but brackets can be used to control precedence. For example:

(/Age > 50 or /Department eq "Accounts") and /Band > 3

Examples of using the process transformation in a topic view specification

The following topic view specification can be used to write a new field (or overwrite an existing one) in the value of the reference topic. It will operate on every JSON topic under the root node a and create new reference topics with the same sub-path under the root node b.

map ?a/ to b/<path(1)> process {set(/Name, "John")};

The following example shows a simple conditional statement that would only generate reference topics under filtered if the value of the field /Price was greater than 50 :

map ?a/ to filtered/<path(1)> process {if "/Price gt 50" continue};

The following example shows a statement that removes some fields if they exist :

map ?a/ to redacted/<path(1)> process {remove(/Salary) ; remove(/Bonus)};

The following shows a more complex statement that would set a field according to the value of the input field /Price :

map ?a/ to tiered/<path(1)> process {if "/Price lt 50" set(/Tier, 1) elseif "/Price gt 50" set(/Tier, 2)};

And the following shows the same as previous but where no tier is set for a price greater than or equal to 80 :

map ?a/ to tiered/<path(1)> process {if "/Price lt 50" set(/Tier, 1) elseif "/Price lt 80" set(/Tier, 2) else continue};

Finally, the following specification performs a calculation on the input topics to produce a modified set of topics :

map ?a/ to annuals/<path(1)> process {set(/AnnualGrossIncome, calc "/MonthlyRate * 12 + /Bonus") ; remove(/MonthlyRate) ; remove(/Bonus)};

Summary

The new process transformation brings new capabilities to topic views that allow them to be used for various use cases such as :

• Augmentation
  Setting new field values in a topic value, possibly conditional upon or calculated from other fields in the input value.
• Filtering
  Generating reference topics only if certain conditions based upon the input value are satisfied.
• Redaction
  Removing fields or structures from a value. Possibly more useful than JSON patch as the transformation does not fail if the items do not exist.

This clause provides new data wrangling capabilities that further sets Diffusion apart from other pub-sub platforms.

The capabilities of the process clause will be extended further in future releases, for example to support floating-point arithmetic, string manipulation, date handling, other operators and more.