Apache Kafka
  • 19 Jul 2022
  • 3 Minutes to read
  • Dark
    Light

Apache Kafka

  • Dark
    Light

Overview

Module Details

Installation LocationCore Module
Step Location N/A
Settings Location System > Settings Kafka Message Queue Settings
Prerequisites
  • An Apache Kafka Account.
  • Configured Apache Kafka application.
  • Apache Kafka Module installed
  • Understanding of Message Queues in Decisions.

The Kafka module provides integration with Apache Kafka to read and write messages to a message queue.  The Message Handler for Kafka can only retrieve a single message at a time. 

To retrieve more than one message a Flow must be configured using a ForEach step.







Kafka Module Settings

  1. Navigate to System > Settings, then either select Kafka Message Queue Settings or right-click its name in the Report and select Edit.

  2. From the Edit Kafka MQSettings window, under Bootstrap Servers, click ADD. From the Add Bootstrap Servers screen, provide the Host and Port for the Server then click OK. 



  3. If Apache Kafka is using SSL, check the Use SSL box, then provide the CA File Path, Cert File Path, and Private Key File Path that correspond to the SSL Certificate. 

  4. If needed, click ADD under Additional Config Options. From here provide the Name and Value pairs, such as the SSL keys. Then, click OK and SAVE.

    Below is an example that will keep the message queue alive when no messages exist.

    NameValue
    socket.keepalive.enabletrue



Adding a Kafka Message Queue 

To connect to a Kafka Message Queue, navigate to System > Integration > Jobs and Events > Message Queues and select the ADD KAFKA QUEUE from the Action Bar. 

Kafka Message Queue Properties

SettingDescription
Display NameThe name which will appear in Decisions. 
Topic The name of the Topic in Kafka the messages will be sent to. 
Show Advanced SettingsDisplays advanced settings that can be used to configure how messages will be sent and received.

Use Custom Connection InfoAllows users to override the Server settings defined from the Edit KafkaMQ Settings window including the ability to add Bootstrap Servers and denote SSL Settings.

BootStrap Servers
Allows users to create a list of servers that can be added and connected to. The Host and Port should match what is running for Apache Kafka. This is typically the host's DNS name or IP Address and the default or user-declared Port number. 

Use SSL If enabled, allows the user to specify the SSL certificate.  

Consumer Group IDConsumer group identifier.

Get Messages from Specific PartitionIf enabled, splits incoming messages from a specific partition. 

Get from PartitionSpecifies which partition to pull the messages from.

Message Handling PreferenceSpecifies how messages will be received and handled. The two options are At Least Once or At Most Once

Message Partition ConfigurationDesignates where an outgoing message will be sent. Currently, there are three options:
  • No key: Sends the message without a key.
  • Post to Specific Partition: This allows users to designate the Partition based on a Partition number.
  • Include Key: Opens and runs a Flow each time a message is sent. This Flow takes in the message, and outputs it as a Byte Array. 

Always Retry After Connection Errors(Dangerous)By design, the Kafka Module will stop and prevent message queues from creating threads if 20 consecutive attempts have failed. If this setting is enabled, messages will continue to create threads even after reaching the attempt count.

Kafka Message Queue Controls

  1. Navigate to System > Jobs and Events > Message Queues.
  2. On the Global Action Bar, select CONTROL > Kafka, then the desired action. 

ControlDescription 
Resync QueueRecreates the Message Queues if necessary and re-synchronizes with the Message Server.
Start QueueStarts processing messages on the Message Queues (if Stopped).
Stop QueueStops processing messages on the Message Queues. The Message Server will hold any unprocessed messages.
Test QueuesReads the number of waiting messages on the Queue and displays a popup with the number of waiting messages. This action helps verify connectivity in a Queue.



Kafka Message Queue Steps

For releases 8.4+ and above, the Kafka module can utilize the Post Message and Post String Message steps listed under Integration > Message Queues. This avoids the need of having to encode and decode a string of bytes when passing a message to a queue. Each step listed below will require a Kafka Queue configured in Decisions in order to be used.


For further information on Integrations, visit the Decisions Forum.

Was this article helpful?