Session Trees – A New Feature for 6.7

Release 6.7 of Diffusion™ introduces a new and exciting feature called ‘session trees‘.

Some History – Routing Topics

(You can skip this section if you are not a current user of routing topics).

Existing users of Diffusion may be familiar with ‘routing topics’. Routing topics have been a part of Diffusion for some time and the problem they solve is the requirement to present different data to different users at the same topic path. A typical use case for this is where users at a different service level might receive different prices, odds, conversion rates, etc. Routing topics allow a routing handler to be called when a client session subscribes to the topic, and the handler can then assign a different topic to provide the data to the routing topic for that client session.

Some of the limitations of routing topics are listed below.

1. A hierarchy of routing topics must be created and maintained. This requires application code to be executed by a control session.
2. Every routing subscription request must be processed by a routing handler. This requires application development and adds latency, and potentially limits scalability.
3. Routing decisions are not persisted for continuous evaluation. If the resolved source topic is removed and recreated, the subscription will be lost and not re-triggered. This could happen quite naturally if the target is maintained by a topic view.
4. Routing decisions are not persisted to the cluster. If a session fails over, the routing handler will be called again for each routing topic.
5. The user is responsible for keeping at least one control client running, hosting a routing handler.
6. The routing handler receives practically no context with which it can make the decision: only the session ID and the selector. Practically, an application must use session properties listeners, or getSessionProperties(), resulting in further delays and bottlenecks.
7. It is difficult for the routing handler to usefully interpret the selector without consulting the topic tree.
8. If a resolved target topic is removed, sessions are unsubscribed but the routing handler is not automatically called again on behalf of those sessions. It is practically impossible for a control client to identify such sessions, and even if it could, there is no API to re-trigger routing.
9. Both the routing topic and target topic specifications determine the behavior. Ownership and “tidies on unsubscribe” are determined by the routing topic specification, but the priority, compression policy, and conflation policy depend on the target topic specification. Also, a routing topic’s specification determines the priority used to replicate the routing topic, but the priority and conflation policy is otherwise ignored.

Existing alternatives to routing topics include:

• The “concierge pattern” ­– a session sends a request to a control session that executes the subscription on its behalf. This fixes the lack of context, and the continuous evaluation (since the session ends up subscribed to a resolved topic selector). However, source topic paths are exposed to the session (which may be undesirable for security/information exposure reasons), and the client application must be written with foreknowledge of the possible source topic paths and map them appropriately. Further, to evolve the server topic tree without recoding the client application requires that a topic view be applied.
• Coding knowledge of the source topic paths directly into the application, and using access control to prevent unwanted exposure. This is simple but has similar problems to the concierge pattern because the client application must be aware of the physical topic tree structure.

Session trees have none of the limitations of routing topics and provide a code-free and far more powerful solution. As routing topics are fully replaced by session trees they have now been deprecated and will be removed in a future release.

What are Session Trees?

A session tree is a virtual view of the topic tree presented to a session by fetch and subscription operations. Custom session trees for different sessions can be configured using declarative rules maintained by the server to meet data security, data optimization, or personalization and localization requirements. Each session can be presented with a unique session tree based upon its session properties.

Branch Mappings

A session tree is produced by applying branch mappings to the topic tree. Branch mappings are organized into branch mapping tables. Each branch mapping table is assigned to a unique path – the session tree branch.

A session tree is composed of session paths. Each session path is mapped via the branch mapping tables to a unique topic path.

A branch mapping table is an ordered list of (session filter, topic tree branch) pairs. For example, the branch mapping table for the session tree branch market/prices might be:

Session filter                           Topic tree branch
=============                            =================
USER_TIER is '1' or $Country is 'DE'     backend/discounted_prices
USER_TIER is '2'                         backend/standard_prices
$Principal is ''                         backend/delayed_prices

With this configuration, if an unauthenticated session (one that matches the $Principal is session filter) subscribes to the session pathmarket/prices/X, and there is a topic bound to the topic pathbackend/delayed_prices/X, the subscription will complete. The session will receive a subscription notification under the session path market/prices/X, together with the topic properties and the value of the topic. The session is unaware that the data originates from a topic bound to a different topic path. If no topic is bound tobackend/delayed_prices/X, the subscription will not resolve and the session will receive no data, even if there is a topic bound to market/prices/X.

Session trees complement the data transformation capabilities of topic views. In our example, the time-delayed time feed at backend/delayed_prices could be maintained by a topic view using the delay by clause.

Branch mappings are persisted by the server and shared across a cluster, in a similar manner to topic views, security stores, and metric collectors. Branch mappings are editable in the management console. New API functions allow branch mapping maintenance by client applications.

In the diagram below, both sessions subscribe to the branch X (using a selector ?X/) but the branch mapping tables are consulted to evaluate the session properties of the sessions. Because session 1 has the property Tier set to 1 it is mapped to the B branch of the topic tree, and session 2 is mapped to the C branch. Both sessions see the topics they are subscribed to as having the prefix X but each is receiving data from a different part of the actual topic tree.

Single Mapping Rule

For a given session and session path, at most one branch mapping table applies.

The applicable mapping is chosen as follows:

• Each branch mapping table with a session tree branch that is a prefix of the session path is considered. For a given table, the first branch mapping with a session filter that matches the session’s properties is the one that applies. A branch mapping table may have no applicable branch mappings for a session.
  
• If there are several such branch mapping tables with a mapping for the session, the one with the longest prefix of the session path applies.
• If no branch mappings table has a branch mapping for the session, the session path is translated to the identical topic path.

Effect on Session Read Operations (Subscribe, Fetch, Time Series Range Query)

Selectors provided by the session are matched against session paths in the session tree. The branch mapping tables are used to translate the session paths to topic paths. The topic paths are translated back to session paths when they are returned to the session (via subscriptions or fetch).

With our example branch mapping table at the source branch market/prices, a tier 2 user subscribing using the path selector >market/prices/fish/cod will be subscribed to the topic backend/standard_prices/fish/cod (assuming the topic exists). When the subscription completes, the session will be passed the session path market/prices/fish/cod, with the topic specification and value of backend/standard_prices/fish/cod.

Fetch and time series range query operations work similarly.

Dynamic re-evaluation of rules

Any changes to the permissions of a session, or topic that a session is accessing through session trees will cause the session tree to be dynamically re-evaluated. The session will be unsubscribed from any topics it no longer has access to and subscribed to any that it now has access to via its current topic selection.

New subscriptions from a session are also re-evaluated against existing branch mapping tables.

If branch mapping tables are changed then the session trees of all affected sessions are also dynamically re-evaluated to apply any changes to mapping rules.

What Can I Use Session Trees for?

Diffusion adapts and enriches data streams and distributes to end users. Session trees allow you to be specific about what each end user gets.

Traditionally, such control might have been achieved by having the knowledge in the client application about what the end user gets, or alternatively, control could be achieved using permissions. Session trees allow the client application to be far simpler, in that the data the client receives is defined by declarative rules within the server. This provides a very-low-code solution to controlling the data each individual client receives. The client session (or to be more accurate the client application) does not even know that it is receiving different data from other sessions for the topics it has subscribed to.

When used in conjunction with forced subscriptions (the SubscriptionControl feature in the API), session trees can be used to present a wholly tailored topic tree to each client session. The client might only subscribe to a single top-level topic and using session trees along with missing topic notifications a full tailored sub-tree can be presented.

The uses of session trees are therefore manifold. Different prices or odds can be presented to each individual user session. They can be used for internationalization or different geographic requirements. Lower-grade clients can be presented with different data flows by mapping to throttled or delayed topics (topic view features). Customer care applications can present different levels of service.

All of this specific topic mapping using a session’s properties can be achieved with practically no special code in the client application.

Defining Branch Mapping Tables

Branch mapping tables may be defined and changed using the client APIs. The tables are persisted in the server and distributed across the cluster. As branch mapping tables are persistent they will continue to apply even when the server is restarted, until they are explicitly removed.

The following Java code shows how to define the branch mapping table for the example given above:-

final BranchMappingTable branchMappingTable =
    newBranchMappingTableBuilder()
        .addBranchMapping(
            "USER_TIER is '1' or $Country is 'DE'",
            "backend/discounted_prices")
        .addBranchMapping(
            "USER_TIER is '2'",
            "backend/standard_prices")
        .addBranchMapping(
            "$Principal is ''",
            "backend/delayed_prices")
        .create("market/prices");

session.feature(SessionTrees.class)
    .putBranchMappingTable(branchMappingTable).get(5, SECONDS);

In terms of permissions, the subscribing session only needs to have permission to the session path it is subscribing to. It does not need permissions to any topics that the branch mapping table maps it to. The creator of a branch mapping table needs MODIFY_TOPIC permission to the session tree branch of the branch mapping table and also the new EXPOSE_BRANCH permission for the topic tree branch of each branch mapping.

Summary

Session trees are a new and powerful feature providing a low-code solution to complex topic mapping requirements. Session trees provide a scalable and distributed solution that perfectly complements other Diffusion features such as topic views, missing topic notifications, and forced subscriptions. Sessions trees fully replace and resolve all of the limitations of the legacy routing topics (which are now deprecated).

Session trees provide an extremely powerful addition to Diffusion’s ability to consume, enrich and deliver event data efficiently and in real time at internet scale.