New Kafka Adapter: Connect Diffusion 6.6 with Apache Kafka

With the release of Diffusion 6.6 Preview 1, a beta version of the new Kafka adapter is now available for on-premise Diffusion.

The Kafka adapter translates data between Diffusion topics and Apache Kafka events. We’ve designed it to make it quick and easy to integrate Kafka with Diffusion.

Kafka is a widely used event streaming tool with high performance, solid scalability and resilience within the data center. Diffusion is efficient in providing the last mile delivery of data across unreliable networks, using delta streaming and reliable session reconnection. The new Kafka adapter acts as a bridge between these two robust systems and enables you to build a complete real-time data distribution solution.

The earlier standalone Kafka adapter for Diffusion used the Kafka Connect framework and was available via GitHub. The new adapter does not use Kafka Connect, and is instead based on Kafka Producers and Consumers.

A version of this new Kafka adapter has been available on Diffusion Cloud since August.

For the on-prem preview release, the adapter is more refined with additional features and improved configuration options. The adapter is included within the Diffusion server installation and can be stopped and started independently from the main server.

Features:

Pub-sub

With the publisher function of the adapter, messages from different Kafka topics can be aggregated in Diffusion and extracted into separate reference topics using topic views or modified using the new topic view insert clause. The newly transformed data can be published to another Kafka topic (using the subscriber feature of the adapter) or to Diffusion clients.

You can publish updates from a specific Diffusion topic path to Kafka. If you prefer, you can map updates from multiple Diffusion topic paths to a single Kafka topic. This is all configurable in the adapter.

Multiple Kafka cluster support

In this version of adapter, users can connect Diffusion with multiple Kafka clusters and transfer messages across them. With this feature, data from different Kafka clusters can be published to Diffusion topics. The adapter can be configured to consume messages from one Kafka cluster and publish messages to others. Here’s a diagram of an example use case:

Kafka security support

You can configure the adapter to connect securely to a Kafka cluster. SSL connection and SASL authentication mechanisms are supported. In this version, for SASL authentication, SASL/GSSAPI (Kerberos) and SASL/PLAIN are supported.

Configurable Kafka clients

Apache Kafka provides various configuration options to configure Kafka consumer and producers. These can be freely added as key-value pairs in the adapter configuration, so you can fine-tune the capability of consumers and producers.

Time series topics

Publishing and subscribing to Diffusion time series topics are supported by the adapter. To publish to a time series topic, use the “mapToTimeSeriesTopic” flag in the configuration.

Monitoring

The adapter uses Prometheus to expose its metrics. In addition to the default JVM level metrics provided by Prometheus, publisher and subscriber specific counters are added, giving you more insight into the messages processed by adapter.

More information

The 6.6 Preview 1 manual has full documentation about how to configure and use the Kafka adapter.

Remember, you can download and install 6.6 Preview 1 now or create a 6.6 service on Diffusion Cloud.

Please let us know what you think about the new adapter. We plan to expand its capabilities in the full 6.6 release and subsequent releases, so please get in touch with your feedback and feature requests.