Fine-Grained Fan-out and Replication of Kafka Event Firehose Between Clusters

Fine-grained fan-out and replication of Kafka event firehose between clusters/sites.

Introduction to Diffusion Real-Time Event Stream through a simple application using Diffusion Cloud and Apache Kafka.

This simple project illustrates real-time replication and fan-out of foreign exchange (fx) event streams from Kafka cluster A to Kafka cluster B, through Diffusion Cloud instance via the use of our Kafka Adapter.

These JavaScript code examples will help you publish fx events on real-time from a front-end app to a Kafka cluster A, consume from it and transform data on-the-fly via our powerful Topic Views feature.

Although we provide a fx data simulation client using JavaScript, to populate Kafka cluster A, this tutorial purely focuses on the no-code solution to deliver event data between remote Kafka sites/clusters where not all the firehose data from one site needs to be replicated/fanned-out to the other.

Fine-grained distribution of Kafka event firehose with Topic Views

kafka-app-L1 introduces the concept of Topic Views, a dynamic mechanism to map part of a server’s Topic Tree to another. This enables real-time data transformation before replicating to a remote cluster as well as to create dynamic data models based on on-the-fly data (eg: Kafka firehose data). This lesson also shows how to use our Kafka adapter to ingest and broadcast fx data using Diffusion Topic Views in order to consume what you need, not all the Kafka stream.

Features used in this lesson

Step 1: Configure Kafka Adapter in Cloud to ingest from Kafka Cluster A
Go to: Diffusion Cloud > Manage Service > Adapters > Kafka Adapter > Ingest from Kafka

Adapters > Kafka Adapter > Ingest from Kafka Config:
- Bootstrap Server > connect to you Kafka cluster A (eg: "kafka-sasl.preprod-demo.pushtechnology")
- Diffusion service credentials > admin, password (use the "Security" tab to create a user or admin account)
- Kafka Topic subscription > the source topic from your Kafka cluster (eg: "FXPairs")
- Kafka Topic value type > we are using JSON, but can be string, integer, byte, etc.
- Kafka Topic key type > use string type for this code example.

Step 2: Check the Kafka stream FXPairs is ingested
Go to: Diffusion Cloud > Manage Service > Console > Topics
We can see the events from FXPairs Kafka topic (from cluster A) are now being published to Diffusion topic path: FXPairs. If there are no new events, it might be because the FXPairs topic has not received any updates from Kafka yet.

Step 3: Create Topic Views using Source value directives

Source value directives use the keyword expand() value directive to create multiple reference topics from a single JSON source topic, or value() directive to create a new JSON value with a subset of a JSON source topic.

Go to: Diffusion Cloud > Manage Service > Console > Topics > Topic Views

We are going to map the topic FXPairs stream (we get from Kafka cluster A) to a new Diffusion Topic View with path: pairs/<expand(/value/pairs,/pairName)>/ where /value/pairs,/pairName is the Kafka payload currency pairName (part of the JSON structure in the FXPairs Kafka topic).

This Topic View will take care of dynamic branching and routing of event streams in real-time, by only sending the specific currency pair from Kafka cluster A, to Kafka cluster B, and not the whole stream.

Topic View Specification for Currency Breakout:

Topic View Specification for Tiers Expansion:

Step 4: Dynamic branching and routing of Kafka events firehose

As new events are coming in from the Kafka cluster A firehose, Diffusion is dynamically branching and routing the currency pairs, on-the-fly when replicating and fan-out to Kafka cluster B.

Note: The topic path will dynamically change as new currency pair values come in.

Go to: Diffusion Cloud > Manage Service > Console > Topics

The following image shows a Topic View for the following specification:

+ pairs

+ tiers

When clicking on the “+” for “tiers” topic tree, the following image shows a Topic View for the following specification:

Suggested: 6 Lessons Using Topic Views

Lesson 1: Mapping Topics

Lesson 2: Mapping Topic Branches

Lesson 3: Extracting Source Topic Values

Lesson 4: Throttling Reference Topics

Lesson 5: Naming Reference Topic With Topic Content

Lesson 6: Changing Topic Properties Of Reference Topics

Step 5: Configure Kafka Adapter in Cloud to broadcast to Kafka cluster B

Go to: Diffusion Cloud > Manage Service > Adapters > Kafka Adapter > Broadcast to Kafka

Adapters > Kafka Adapter > Broadcast to Kafka Config:
- Bootstrap Server > connect to you Kafka cluster A (eg: "kafka-plain.preprod-demo.pushtechnology")
- Diffusion service credentials > admin, password (use the "Security" tab to create a user or admin account)
- Topic subscription > the source topic from Diffusion to be broadcasted to Kafka cluster B (eg: from "tiers/2/GBP-USD" to "FXPairs.tier2.GBP.USD")
- Topic value type > we are using JSON, but can be string, integer, byte, etc.
- Kafka Topic key type > use string type for this code example.

Pre-requisites

• Download our code examples or clone them to your local environment:

• A Diffusion service (Cloud or On-Premise), version 6.5.0 or greater. Create a service here.

• Follow our Quick Start Guide and get your service up in a minute!

FX Data Generator

Really easy, just open the ‘index.html‘ file locally and off you go!

Download the Code Now

Get familiar with our SDK, our user friendly dashboard and all the power of our platform. Start Now!

Watch our Webinar

Watch our webinars introducing concepts of your Kafka Adapter, Diffusion Dashboard, Console and Data Wrangling.

Thank you!