Home

Older Versions

Apache Kafka

26 May 2022
2 Minutes to read

Print
Share
Dark
Light

This documentation version is deprecated, please click here for the latest version.

Apache Kafka

Updated on 26 May 2022
2 Minutes to read

Print
Share
Dark
Light

Article summary

Did you find this summary helpful?

Thank you for your feedback

Overview

Module Details
Installation Location	Core Module
Step Location	N/A
Settings Location	System -> Settings Kafka Message Queue Settings
Prerequisites	An Apache Kafka Account. Configured Apache Kafka application. Apache Kafka Module installed Understanding of Message Queues in Decisions.

The Kafka module provides integration with Apache Kafka to read and write messages to a message queue. The Message Handler for Kafka can only retrieve a single message at a time.

To retrieve more than one message a Flow must be configured using a ForEach step.

Kafka Module Settings

Navigate to System > Settings, then either select Kafka Message Queue Settings or right-click its name in the Report and select Edit.
From the Edit Kafka MQSettings window, under Bootstrap Servers, click ADD. From the Add Bootstrap Servers screen, provide the Host and Port for the Server then click OK.
If Apache Kafka is using SSL, check the Use SSL box, then provide the CA File Path, Cert File Path, and Private Key File Path that correspond to the SSL Certificate.
If needed, click ADD under Additional Config Options. From here provide the Name and Value pairs, such as the SSL keys. Then, click OK and SAVE.
Name Value
socket.keepalive.enable true

Name	Value
socket.keepalive.enable	true

Adding a Kafka Message Queue

To connect to a Kafka Message Queue, navigate to System > Integration > Jobs and Events > Message Queues and select the ADD KAFKA QUEUE from the Action Bar.

Kafka Message Queue Properties

Setting			Description
Display Name			The name which will appear in Decisions.
Topic			The name of the Topic in Kafka the messages will be sent to.
Show Advanced Settings			Displays advanced settings that can be used to configure how messages will be sent and received.
	Use Custom Connection Info		Allows users to override the Server settings defined from the Edit KafkaMQ Settings window including the ability to add Bootstrap Servers and denote SSL Settings.
		BootStrap Servers	Allows users to create a list of servers that can be added and connected to. The Host and Port should match what is running for Apache Kafka. This is typically the host's DNS name or IP Address and the default or user-declared Port number.
		Use SSL	If enabled, allows the user to specify the SSL certificate.
	Consumer Group ID		Consumer group identifier.
	Get Messages from Specific Partition		If enabled, splits incoming messages from a specific partition.
		Get from Partition	Specifies which partition to pull the messages from.
	Message Handling Preference		Specifies how messages will be received and handled. The two options are At Least Once or At Most Once.
	Message Partition Configuration		Designates where an outgoing message will be sent. Currently, there are three options: No key: Sends the message without a key. Post to Specific Partition: Allows users to designate the Partition based on a Partition number. Include Key: Opens and runs a Flow each time a message is sent. This Flow takes in the message, and outputs it as a Byte Array.
	Always Retry After Connection Errors(Dangerous)		By design, the Kafka Module will stop and prevent message queues from creating threads if 20 consecutive attempts have failed. If this setting is enabled, messages will continue to create threads even after reaching the attempt count.

Kafka Message Queue Controls

Navigate to System > Jobs and Events > Message Queues.
On the Global Action Bar, select CONTROL > Kafka, then the desired action.

Control	Description
Resync Queue	Recreates the Message Queues if necessary and re-synchronizes with the Message Server.
Start Queue	Starts processing messages on the Message Queues (if Stopped).
Stop Queue	Stops processing messages on the Message Queues. The Message Server will hold any unprocessed messages.
Test Queues	Reads the number of waiting messages on the Queue and displays a popup with the number of waiting messages. This action helps verify connectivity in a Queue.

Was this article helpful?