myEsn2E9

Core Concepts#

RabbitMQ is an overall producer-consumer model that is primarily responsible for receiving, storing, and forwarding messages.

Overall Architecture of RabbitMQ

• Producer and Consumer
• Message Composition:
  • Message Header (Label):
    • routing-key
    • priority
    • delivery-mode
  • Message Body (payload)
• Exchange: Producers send messages to the Exchange, which then routes the messages to one or more queues. If the routing fails, the message is returned to the producer or discarded directly.

  • Process: Publish Message + RoutingKey → Exchange → Binding Key → Queue > Consume
    

  • Common Types:

    • direct (default): Routes messages to a Queue that has a BindingKey matching the RoutingKey. Commonly used for handling tasks with priorities.

      

    • fanout: Routes messages to all queues bound to it. Commonly used for broadcasting messages.

    • topic: Routes messages to queues with BindingKeys and RoutingKeys that match (exact match or wildcard match).

      • BindingKeys and RoutingKeys are strings separated by periods.
      • BindingKeys can contain two types of wildcard matching strings: "*" (matches one word) and "#" (matches zero or more words).
        

    • headers (not recommended): Matches based on the headers property in the message content. Poor performance, not practical, and rarely used.

• Queue: Container for messages, where a message can be placed in one or more queues.
  • Consumption: Supports one or more consumers (average Round-Robin) subscribing to the same queue.
• Binding: Binds a Queue to an Exchange using a BindingKey.
• Broker: A RabbitMQ Broker can be seen as a RabbitMQ service node or instance.
• Dead Letter Exchange: When a message becomes a dead message in a queue, it is sent to the dead letter exchange.
  • Reasons for becoming a dead message:
    • Message is rejected (Basic.Reject / Basic.Nack) with requeue = false
    • Message TTL expires
    • Queue is full
• Delayed Queue: Messages are consumed after a specific time.
  • Implementation methods:
    1. Using RabbitMQ's Dead Letter Exchange and message's time to live (TTL).
    2. Using the plugin developed for RabbitMQ 3.5.7+ (rabbitmq-delayed-message-exchange).
• Priority Queue: Implemented in RabbitMQ 3.5.0+ using the x-max-priority parameter. This parameter is not meaningful when the consumption speed is greater than the production speed and the broker has no backlog.
• Message Transmission: Based on channels established over TCP connections, with unlimited quantity. Each channel has a unique ID in RabbitMQ, corresponding to a thread.
• Message Reliability:
  • Producer to RabbitMQ: Transaction mechanism or Confirm mechanism. Note: Transaction mechanism and Confirm mechanism are mutually exclusive and cannot coexist, as it will cause RabbitMQ to throw an error.
  • RabbitMQ itself: Durability, regular cluster, mirrored cluster
  • RabbitMQ to Consumer: basicAck mechanism, dead letter queue, message compensation mechanism
• Message Ordering:
  • Split into multiple queues, with each queue having one consumer (reduces throughput)
  • Consumers do not consume messages directly. Instead, they hash based on a key value in the message (e.g., ID, meaning multiple messages with the same ID), and then group them into different in-memory queues based on the hash value. Finally, each group is handed over to a different worker or sub-thread for processing (consumed using multi-threading within the consumer).
• High Availability:
  • Regular cluster: Data is stored on the master node, and the slave nodes only synchronize the metadata of the queues from the master node, not the messages.
  • Mirrored cluster: Data is fully synchronized between the master and slave nodes.