Apache Kafka

Overview

The Kafka module provides the tools needed to connect to a Kafka Message Stream. Kafka is an open source event streaming services that is mainly used as a message queue. 

With the module, Decisions can setup and connect to a Kafka Queue.

The Kafka module can utilize the Post Message and Post String Message steps listed under Integration > Message Queues. 

Using these steps avoids having to encode and decode a string of bytes when passing a message to a queue. Each step listed below will require a Kafka Queue configured in Decisions to be used.

This document provides a step-by-step guide to setting up Confluent Kafka. However, any version of Kafka is supported. Confluent is only used as an example.

Kafka Module Settings

The below steps are instructions for viewing and updating the module settings.

1. Navigate to System > Settings, then either select Kafka Message Queue Settings or right-click its name in the Report and select Edit.
   
2. From the Edit Kafka MQSettings window, under Bootstrap Servers, click ADD. From the Add Bootstrap Servers screen, provide the Host and Port for the Server then click OK. 

   

   Setting	Description
   Host	The host IP or name of the server hosting Kafka.
   Port	The Port number used by Kafka on its host machine.

   
   

3. If Apache Kafka is using SSL, check the Use SSL box, then provide the CA File Path, Cert File Path, and Private Key File Path that correspond to the SSL Certificate. 

   Setting	Description
   Use SSL	Uses SSL to establish a connection to Kafka. An SSL certificate must be supplied and accessible on the machine hosting Decisions.
   CA File Path	The path to the certificate store.
   Cert File Path	The physical file path to the SSL certificate.
   Private Key File Path	The file path to a file containing the private keys used by the certificate.

   
   
   
4. If needed, click ADD under Additional Config Options. From here provide the Name and Value pairs, such as the SSL keys. Then, click OK and SAVE. Below is an example that will keep the message queue alive when no messages exist.

   Name	Value
   socket.keepalive.enable	true

   

   
   

Adding a Kafka Message Queue

To connect to a Kafka Message Queue, navigate to System > Jobs and Events > Message Queues and select the ADD KAFKA QUEUE from the Action Bar. 

Kafka Message Queue Properties

Using Confluent Kafka

Confluent is a cloud-based hosting service created by the original developers of Kafka. It provides a simple way to create and set up a Kafka message broker without using a container. To use this service, an account with Confluent must be created. For more information on setting up Confluent see their quick start guide.

1. Register and signup to access the Confluent panel.
2. Create or use an existing environment within Confluent.
3. Once in an environment, create a Topic in Confluent Kafka by navigating to Topics, then select the Add Topic button.
4. Provide a name and the partition size. Once added, select the Create with Defaults button. A new topic will be created.

Connecting to Decisions

Once a Topic is created, configuration information such as the Bootstrap Server name will be needed. In addition, the following values, security.protocol, sasl.mechanisms, sasl.username, and sasl.password are needed to connect the queue to Decisions.

To locate these values in Confluent:

1. Select Clients, then create a new Client
   
2. Under Choose your language, Select C#
   
   
3. In the next step, click on the Create a new Kafka Cluster API Key button. This will populate the following box with the connection information.
   
   
4. Save the API Key
   
   
5. Once the key is generated, the config file should update. These will be the values that will be entered into Decisions. Save this section into a new file for future reference.
6. In Decisions, navigate to System > Settings > Kafka Module Settings
7. Select ADD for Bootstrap Servers. Enter the host name and port from the bootstrap.servers entry from the configuration file.
8. Under Additional Config Options, Select Add and create entries for security.protocol, sasl.mechanisms, sasl.username, sasl.password entries from the configuration file.
9. Click OK and Save the settings.
10. Right click the module settings, select Manage > Resync Queue to resync the queue.

Feature Changes