< Back to Tutorials

Extend Kafka Topic Modeling with Diffusion

Diffusion Real-Time Event Stream

Introduction to Diffusion Real-Time Event Stream through a simple application using Diffusion Cloud and Apache Kafka.

A set of simple projects, illustrating production and consumption of foreign exchange (fx) data streams to and from Kafka clusters and Diffusion Cloud instance via the use of our Kafka Adapter.

These JavaScript code examples will help you publish fx data on real-time from a front-end app to a Kafka cluster, consume from it and transform data on-the-fly via our powerful Topic Views feature. You can also use other programming languages from our SDKs, including iOS, Android, C, .NET, and more.

Lesson 1: Publish event streams to Kafka cluster

Lesson 1 code example introduces the concept of our Kafka Adapter and its configuration, to consume fx topic data from your Diffusion server and publish it to Kafka cluster in specific Kafka topic.

Kafka Broadcast Diagram

Lesson 2: Fine-grained distribution of Kafka event firehose with Topic Views

Lesson 2 code example introduces the concept of Topic Views, a dynamic mechanism to map part of a server’s Topic Tree (data structure) to another. This enables real-time data transformation before sending it to subscribed clients as well as to create dynamic data models based on on-the-fly data (eg: Kafka firehose data).

Kafka Ingest Diagram

Lesson 3: Security with Topic-level access control

Lesson 3 code example introduces the concept of Security and topic path permissions for fine-grained security management of your data structure. It also shows how to Subscribe to fx data using Diffusion Topic Views in order to consume what you need, not all the Kafka stream.

Pre-requisites

  • Download our code examples or clone them to your local environment:
git clone https://github.com/diffusiondata/tutorials/
  • A Diffusion service (Cloud or On-Premise), version 6.5.0 or greater. Create a service here.

Create Service

Setup

Make sure to add Diffusion library to your code. For JavaScript, we have added the following line in our ‘public/kafka2diffusion.html‘ and ‘public/diffusion2kafka.html‘:

src=’https://download.diffusiondata.com/clients/6.5.1/js/diffusion-6.5.1.js’

Update ‘public/js/producerApp.js‘ and ‘public/js/subscriberApp.js‘ to the hostname of your Diffusion Cloud service, which you can find in your service dashboard. You can also leave the default values and connect to our sandbox service:

  • host: host (“kafkagateway.us.diffusion.cloud” by default)
  • user: ‘user’
  • password: ‘password’

Execution

Really easy, just open the ‘index.html‘ file locally and off you go!

The code in action

(Click to watch the video)

Video Tutorial

Download the Code Now

Get familiar with our SDK, our user friendly dashboard and all the power of our platform.
Start Now!

Watch our Webinar

Tuned our webinar introducing the basic concept of your Diffusion dashboard, console and Data Wrangling.

Thank you!

View all Tutorials