Diffusion 6.1 introduces powerful new features for dynamic topic management

In Push Technology’s ongoing effort to simplify data management, optimization, and integration for companies developing business critical applications, Diffusion 6.1 introduces a pair of powerful new features: automatic topic removal and topic ownership.

Introduction

Diffusion uses topics to deliver data to clients via a publish and subscribe interaction model. A Diffusion topic is an identified stream of data values to which a client can subscribe. Whenever a topic is updated with a new value, all subscribers of the topic are automatically sent the new value. Clients can subscribe to a selection of topics to receive multiple streams of data, and similarly, back-end services can create and remove topics dynamically to reflect structural changes to a system’s data model. Because Diffusion pushes new data to subscribers instead of being polled for updates, once a topic has been removed no further network traffic is involved – clients are simply informed of the removal, and can choose to handle this event in an appropriate manner.

In contrast to traditional publish and subscribe messaging systems, Diffusion topics are lightweight; are fine-grained, representing discrete data items; and are frequently created and removed. Large scale Diffusion applications frequently use tens of millions of topics.

Automatic Topic Removal

Automatic topic removal allows an optional removal policy to be set for a topic using the new REMOVAL topic property. A removal policy is a declarative expression that combines conditions that can include elapsed time; the number of subscribers to the topic; the last time the topic was updated; and the session properties of all clients connected  to the server. The Diffusion server monitors these conditions and automatically removes the topic when the policy is satisfied.

Automatic topic removal considerably simplifies the common tasks of dynamically removing redundant or outdated parts of an application’s data model. Sometimes it is known that a topic data is only valid for a fixed time period. A low number of subscriptions indicates that a topic is no longer of interest to clients. If updates have stopped, it is a strong signal that the source of the information is stale or unavailable, and removing the topic passes this information on to subscribers. An application can use session properties to label different types of client, and the lack of a session of a particular type may also indicate that it is no longer worth maintaining a topic.

Previously, application housekeeping functions would have to be developed and registered with a separate scheduler. Subscription counts could be monitored using topic event listeners, but these are only notified of zero-subscriber threshold events (so cannot trigger removal on fewer than 100 subscribers, for example). Session properties could be monitored through a session properties listener. Monitoring the update rate of a topic required some ingenuity; for example, subscribing to the topic and using client-side timers.

Automatic topic removal can be applied to all of these scenarios. Let’s illustrate the power of removal policies with some examples:

The automatic topic removal feature replaces topic event listeners and the removeTopicsWithSession method. Both topic event listeners and removeTopicsWithSession are deprecated from Diffusion 6.1.

Topic Ownership

Topic ownership provides enhanced security functionality, giving fine-grained access control on a per-topic basis. Sometimes a “private topic” is useful to communicate with a single user. The user might use several devices, have multiple sessions open in different browser tabs, or frequently close and re-open sessions. In each case, a topic is useful to provide a consistent, fresh view across the different sessions. It is important not to expose the topic to other users, and so we want the server to enforce access control. However, per-topic adjustments to the security store are cumbersome, even using the security control APIs, and do not scale well.

Some Diffusion users address this problem with a suitably privileged back-end control session, using the client control API to subscribe user sessions to session-private topics. The control session either uses a session properties listener to detect new user sessions, or relies on the user session sending an application message to the control session on start up. This works, but is a custom solution that must be written and maintained.

Topic ownership provides a different answer. The new OWNER topic property can be used to associate a topic with a security principal. A session authenticated with the security principal is automatically granted the READ_TOPIC, MODIFY_TOPIC, and UPDATE_TOPIC permissions for the topic, in addition to any permissions granted by the security store. All other sessions only have the permissions granted by the security store – typically the security store will be used to grant control sessions appropriate privileges. A control session can create private topics owned by specific principals. Or, if it has the permission to create topics, a user session can create a private topic owned by its principal.

Since topic ownership also grants UPDATE_TOPIC, it supports other private topic use cases. For example, a topic might be created that allows a user to broadcast status information to other users.

Automatic topic removal and topic ownership are designed to work together. We expect it will be common to use a removal policy to remove private topics. For example, the following Java example shows the creation of a topic private/john that is owned by the security principal john and will be removed if an hour passes during which there is no longer a session authenticated with that security principal.

TopicControl topicControl = session.feature(TopicControl.class);

TopicSpecification specification = topicControl
  .newSpecification(TopicType.JSON)
  .withProperty(OWNER, "$Principal is 'john'")
  .withProperty(REMOVAL, "when no session has '$Principal is \"john\"' for 1h");

topicControl.addTopic("private/john", specification);

Automatic topic removal and topic ownership are supported by all of the client SDKs, and are compatible with topic replication.