About Messaging
  • 13 Mar 2024
  • 5 Minutes to read
  • Dark
    Light

About Messaging

  • Dark
    Light

Article summary

Overview

Effective communication is a fundamental requirement with Software Systems. Applications need to exchange information seamlessly and efficiently to meet increasing demands and complexities. Messaging serves as the conduit for data transmission among different software components or systems. While synchronous communication ensures immediate interactions, asynchronous communication enables components to function independently. However, managing asynchronous communication complexity can be challenging. This is where message queues emerge as a solution, optimizing the flow of messages. 

Message queues enable asynchronous communication, boosting scalability and reliability. Asynchronous communication frees applications from waiting for immediate responses, which is essential for tasks needing time-consuming processing or when components are briefly unavailable. Message queues act as intermediaries, letting apps deposit messages for later processing. They uncouple sender and receiver, ensure efficient message handling, and guard against traffic spikes.



What changed from V8 to V9?

Version 8

Version 9

Message Queues were configured at the System level and were visible across the platform.

Message Queues are now Project-level and are visible only within the Project.


Message Queues and Decisions

Through modules, Decisions provides support and integration for popular message brokers. These modules allow Decisions to interact with these applications as either a Producer or Consumer and provide other functionalities such as stopping/starting a queue. Below is a list of currently supported and available message queue modules:

Module
Example Use Case
Advantages
  • Distributed Systems: SQS is well-suited for building distributed systems where components need to communicate asynchronously while remaining decoupled.
  • Batch Processing: It's useful for handling large-scale batch processing tasks, such as data transformation, image resizing, and more.
  • Managed Service: Amazon SQS is fully managed by AWS, handling infrastructure setup, scaling, and maintenance, allowing developers to focus on application logic. 
  • Reliability: Offers redundant storage across multiple availability zones, ensuring high availability and durability of messages. 
  • Flexible Scaling: Automatically scales to handle varying message loads without manual intervention.
  • Event Notifications: SNS is great for broadcasting event notifications to multiple recipients, like email, SMS, or other services. 
  • Fan-Out Architectures: It's useful for fan-out architectures where a single message needs to be sent to multiple subscribers.
  • Push and Publish-Subscribe: Allows messages to be pushed to subscribers in a publish-subscribe model, reducing the need for polling.
  • Diverse Message Formats: Supports multiple message formats, including JSON, XML, and plain text.
  • Integration: Integrates well with other AWS services, allowing easy orchestration of workflows.
  • Complex Routing: ActiveMQ's support for various message exchange patterns is beneficial for scenarios requiring complex routing and filtering. 
  • JMS Integration: It's a good choice for Java-based applications that require integration with the Java Message Service (JMS) API.
  • Open Source: ActiveMQ is an open-source message broker with a strong community and wide adoption. 
  • Flexible Protocols: Supports multiple protocols such as AMQP, STOMP, and MQTT, providing flexibility in integration. 
  • Persistence: Offers durable message storage, ensuring messages aren't lost during system outages.
  • Hybrid Cloud Scenarios: Suitable for scenarios that involve hybrid cloud setups, enabling communication between on-premises and cloud-based applications. 
  • Order Processing: Useful for scenarios where maintaining the order of messages is critical, like order processing systems.
  • Azure Integration: Seamlessly integrates with other Azure services and provides authentication and authorization mechanisms. 
  • Advanced Features: Offers features like message deferral, scheduling, and dead-letter queues for handling complex messaging scenarios. 
  • Partitioning: Supports partitioning to distribute message processing load across multiple instances.
  • Real-time Data Streams: Kafka excels in managing real-time data streams and event-driven architectures. 
  • Log Aggregation: Useful for aggregating logs from various services and applications for monitoring and analysis.
  • High Throughput: Kafka is designed for high throughput and low latency, making it suitable for processing large volumes of data. 
  • Scalability: Scales horizontally across clusters to handle growing data demands. 
  • Fault Tolerance: Provides replication and fault tolerance to ensure data durability and availability.
  • Task Queues: Ideal for distributing tasks among workers, ensuring efficient task processing and load balancing. 
  • IoT Applications: Suitable for Internet of Things (IoT) applications where devices need to communicate with backend services.
  • Versatile Routing: Offers advanced message routing patterns such as direct, topic, and header exchanges. 
  • Ease of Use: RabbitMQ is known for its ease of setup and configuration, making it a popular choice for developers. 
  • Language Agnostic: Provides client libraries for various programming languages, allowing seamless integration.
MSMQ
  • Windows Ecosystem: Well-suited for Windows-centric environments and applications due to tight integration with the Windows operating system. 
  • Offline Scenarios: Useful for offline scenarios where messages can be queued for delivery when the recipient is back online.
  • Reliability: Offers reliable message delivery and ensures that messages aren't lost, even in the event of system failures. 
  • Transactional Support: Supports transactional messaging, where multiple operations are grouped together as a single transaction. 
  • Security: Integrates with Windows security mechanisms, providing robust security features.
  • Microservices Communication: Suitable for microservices architectures where efficient and reliable communication between services is crucial. 
  • Financial Services: Used in the financial industry for high-speed messaging and event streaming in trading platforms.
  • Ultra-Low Latency: Provides ultra-low latency messaging, making it a preferred choice for high-performance applications. 
  • Global Messaging: Offers capabilities for multi-cloud and hybrid cloud deployments, ensuring global messaging connectivity. 
  • Message Routing: Supports advanced message routing and filtering, enabling complex messaging patterns.

Posting Messages to a Queue

Think of posting messages like sending emails to a virtual inbox. Imagine an app notifying users about a new article. Instead of making users wait, the app posts messages to a queue. Just as emails sit in your inbox until you're ready to read them, messages wait in the queue until they're processed. This way, apps keep running smoothly, users stay engaged, and no one's left waiting.

As a producer, use the Post Message step to send these messages to a connected queue. This allows Flows to be configured and define payloads for messages sent to a queue. These Flows can be added to a scheduled job to run requests on a daily basis.


Message Queue Handlers and Active Flow Count

When utilizing Decisions as a consumer, messages can be processed from a queue by creating a Message Handler. These handlers run on specialized threads, ensuring that each message is given the appropriate attention. Active Flow Count determines the number of concurrent message processors. Increasing this count speeds up message processing but requires more resources. Higher counts boost throughput but may strain the system. Finding the right balance is crucial for optimizing performance and resource utilization. This setting is visible through the desired Designer Folder > Settings > Jobs & Events.

Asynchronous Processing and Scalability

Each message triggers the creation of a dedicated Flow, essentially a set of instructions for handling that specific message. The Active Flow Count controls the maximum number of these Active Flows at any moment. This architecture promotes asynchronous processing, allowing multiple messages to be concurrently handled within a single Decisions instance. The number of active Flows can scale to match the available system resources. For greater processing power, Decisions can be configured in a clustered environment. This setup distributes the processing load across multiple nodes, optimizing message handling similarly to how interval jobs are distributed.


Was this article helpful?