Apache Kafka

Overview

Apache Kafka is a messaging system that can be installed as a Module that allows Decisions the ability to integrate the application to application communication. Installing the Module provides a number of Kafka Message Queue Settings that may be configured post-installation.

Configure Apache Kafka Module

This example will demonstrate how to configure the Apache Kafka Module.

1. Navigate to System > Settings and click Kafka Message Queue Settings.
   
2. From the Edit Kafka MQSettings window, under Bootstrap Servers, click Add New. From the Add Bootstrap Servers screen, provide the Host and Port for the Server then click OK. 
   The Bootstrap Servers setting allows users to create a list of Bootstrap Servers that can be added and connected to. The Host and Port should match the numbers of the device running Apache Kafka. This is usually the DNS name or IP Address of the Host and the default or user-declared Port number. 

   
   

3. If Apache Kafka is using SSL, check the Use SSL box, then provide the CA File Path, Cert File Path, and Private Key File Path that correspond to the SSL Certificate. 
   To learn about generating a certificate, see  Generating Certificates with OpenSSL. 
   
   

   

4. If needed, click Add New under Additional Config Options. Provide the Name and Value pairs that are passed into the Confluent.Kafka library and click OK.
   To learn more about the Confluent.Kafka Consumer configuration options, go to https://docs.confluent.io/platform/current/installation/configuration/consumer-configs.html.

   
   

5. Once all necessary options are correctly configured, click SAVE.

Add Kafka Message Queue 

This example will demonstrate how to configure a Kafka Message Queue.

1. Navigate to System > Jobs and Events > Message Queues and click ADD KAFKA QUEUE.
   
2. From the Add Kafka Queue window, provide the desired Display Name and enter a Topic that exists on the Server.
   
3. Check the Show Advanced Settings option. Then if needed, check Use Custom Connection Info and define the parameters. 
   Use Custom Connection Info allows users to override the Server settings defined from the Edit KafkaMQ Settings window including the ability to add Bootstrap Servers and denote SSL Settings. 

   
   

4. Check Get Messages from Specific Partition to allow the splitting of incoming messages. Specify the number in Get from Partition to set which Partition messages are retrieved from another user's Message Queue.
   
5. Select the desired option under Message Handling Preference. 
   The options available are At Least Once and At Most Once. 

   
   

6. Select the desired Message Partition Configuration from the dropdown menu. 
   This setting designates where an outgoing message is sent.
   The options included are:

   • No key: sends the message without a key.
   • Post to Specific Partition: Allows users to designate the Partition based on a Partition number.
   • Include Key: Opens and runs a Flow each time a message is sent (_keyflow). This Flow takes in the message, and outputs it as a Byte Array. 

   
   

7. Once configured, click OK.

Controlling the Message Queue 

This example will demonstrate how to control the Message Queue.

1. Navigate to System > Jobs and Events > Message Queues.
2. Click CONTROL. Select Kafka and select the desired action. 
   The available CONTROL options are as follows:

   • Resync Queues: Recreates the Message Queues if necessary and re-synchronizes with the Message Server.
   • Start Queues: Starts processing messages on the Message Queues (if Stopped).
   • Stop Queues: Stops processing messages on the Message Queues. The Message Server will hold any unprocessed messages. 
   • Test Queues: Reads the number of waiting messages on the Queue and displays a popup with the number of waiting messages. This action helps verify connectivity in a Queue.