Kafka vs RabbitMQ Decision

How do Kafka and RabbitMQ differ?

Kafka and RabbitMQ are both message brokers, but they have fundamentally different architectures that make them suitable for different use cases. Choosing the wrong one creates friction that compounds as the system scales.

Core Architectural Difference

Kafka: Log-Based

Kafka stores messages as an append-only, immutable log on disk. Consumers maintain their own offsets (position in the log) and can read at their own pace, read the same message multiple times, or replay from any position.

• Messages are retained for a configurable period (hours to forever), regardless of whether they were consumed
• Multiple independent consumer groups can read the same topic independently
• Consumers control their own progress - no broker-side acknowledgement state per-consumer

RabbitMQ: Queue-Based

RabbitMQ stores messages in queues. Once a consumer acknowledges a message, it is deleted. The broker tracks delivery state per-consumer.

• Message is delivered once, consumed once (traditional queue semantics)
• Rich routing: exchanges route messages to queues based on routing keys, headers, or patterns (direct, topic, fanout, headers exchange types)
• Consumer-push model: broker pushes messages to consumers

Comparison

Feature	Kafka	RabbitMQ
Message retention	Persistent (configurable TTL)	Deleted on acknowledgement
Replay	Yes - rewind to any offset	No - once consumed, gone
Consumer groups	Yes - independent offsets	Competing consumers (shared queue)
Throughput	Very high (1M+ msg/sec)	High (10K-100K msg/sec)
Ordering	Guaranteed per-partition	Per-queue (single consumer)
Routing	Topic-based, partition key	Flexible exchange types
Protocol	Custom Kafka protocol	AMQP, MQTT, STOMP
Complexity	Higher operational overhead	Simpler to operate

When to Choose Kafka

✅ Event streaming at high throughput (>100K msg/sec) ✅ Multiple independent consumer groups need the same events ✅ Need to replay historical events (event sourcing, data pipelines) ✅ Log-based architecture: audit trail, CDC, stream processing ✅ Integration with Apache Flink/Spark for stream processing

When to Choose RabbitMQ

✅ Traditional work queues (task distribution to workers) ✅ Complex routing logic (route to queue based on message content) ✅ Smaller scale, simpler operational requirements ✅ Existing AMQP infrastructure ✅ RPC pattern (request/reply) ✅ When message must be deleted after processing (don't want consumers missing ACKs to replay stale work)

Consumer Groups (Kafka)

A consumer group allows multiple consumers to cooperate on processing a topic:

• Each partition in the topic is assigned to exactly one consumer in the group
• This gives parallelism within a group without duplicate processing
• Multiple separate consumer groups each get a full copy of all messages

Example: orders topic consumed by 3 groups: inventory-service, notification-service, analytics-service - each independently processes every order event.

Ordering Guarantees

• Kafka: ordered within a partition; no ordering guarantee across partitions. Use a consistent partition key (e.g., order_id) to ensure all events for one order go to the same partition.
• RabbitMQ: ordered within a queue with a single consumer. Multiple consumers on the same queue break ordering.

Interview Angle

For high-throughput event streaming with multiple consumers and replay -> Kafka. For work queue, complex routing, simpler setup -> RabbitMQ. The key differentiator to articulate: Kafka retains messages (replay), RabbitMQ deletes on ACK (traditional queue). Ask the interviewer about replay requirement, number of consumers, and throughput - the answer determines the choice.