Kafka Interview Questions and Answers
22 hand-picked Kafka interview questions with
detailed answers. Open the interactive version above to search, filter
by difficulty, run code, bookmark questions and track your progress.
What is Apache Kafka and what problems does it solve?
Kafka is a distributed event streaming platform — a high-throughput, fault-tolerant, durable log for publishing and subscribing to streams of events. Use cases:
- Event streaming — real-time data pipelines between services.
- Message queuing — decouple producers and consumers.
- Event sourcing / CQRS — durable log of all state changes.
- Log aggregation — collect logs/metrics from many sources.
Unlike traditional message brokers, Kafka retains messages (configurable retention) and allows consumers to replay history.
Explain topics, partitions, and brokers.
- Broker — a Kafka server that stores data and serves clients. A cluster has multiple brokers for fault tolerance.
- Topic — a named category/feed of messages (like a table in a DB).
- Partition — a topic is split into partitions for parallelism. Each partition is an ordered, immutable sequence of messages.
- Offset — a unique sequential ID for each message within a partition.
More partitions = more parallelism (more consumers can read concurrently).
How does a Kafka producer work?
The producer sends records to a topic. Key config:
- key — determines partition (hash of key % num partitions). Same key → same partition.
- acks — durability guarantee:
0 (fire-and-forget), 1 (leader ack), all (all ISR replicas ack). - retries — automatic retry on transient failures.
- idempotence — exactly-once per partition (no duplicates on retry).
Producer batches messages for efficiency and compresses (snappy, lz4, zstd).
Properties props = new Properties();
props.put("acks", "all");
props.put("enable.idempotence", true);
Producer<String, Order> producer = new KafkaProducer<>(props);
producer.send(new ProducerRecord<>("orders", orderId, order));
What is a consumer group?
A consumer group is a set of consumers that jointly consume a topic. Kafka assigns each partition to exactly one consumer in the group — enabling parallel consumption while avoiding duplicate processing within the group.
Rules:
- Consumers in the same group share the work (load balancing).
- Consumers in different groups each get all messages (pub/sub).
- Max useful consumers per group = number of partitions.
props.put("group.id", "order-processor");
props.put("auto.offset.reset", "earliest");
KafkaConsumer<String, Order> consumer = new KafkaConsumer<>(props);
consumer.subscribe(List.of("orders"));
What are offsets and how are they committed?
An offset is the consumer's position in a partition log. Committing an offset tells Kafka "I've processed up to here."
- Auto-commit — periodic background commit (
enable.auto.commit=true). Risk: message processed but crash before commit → reprocessed; commit before processing → lost on crash. - Manual commit — commit after successful processing.
commitSync() (blocking) or commitAsync().
Offsets are stored in the internal __consumer_offsets topic.
while (true) {
ConsumerRecords<String, Order> records = consumer.poll(Duration.ofMillis(100));
for (var record : records) {
process(record.value());
}
consumer.commitSync(); // after batch processed
}
How does Kafka replication work?
Each partition has one leader and zero or more followers (replicas). Producers and consumers talk to the leader; followers replicate data.
ISR (In-Sync Replicas) — followers caught up with the leader. acks=all waits for all ISR replicas. If a leader dies, a follower from ISR is elected as the new leader.
Set replication.factor=3 in production for fault tolerance.
At-most-once, at-least-once, and exactly-once semantics.
| Semantic | How | Tradeoff |
|---|
| At-most-once | Commit offset before processing | Messages may be lost on crash |
| At-least-once | Commit offset after processing | Messages may be reprocessed (duplicates) |
| Exactly-once | Idempotent producer + transactional consumer | Higher latency, more complex |
Most apps use at-least-once + idempotent consumers (deduplicate by message ID).
props.put("enable.idempotence", true);
props.put("transactional.id", "order-tx");
producer.initTransactions();
Kafka vs RabbitMQ — when to use each?
| Kafka | RabbitMQ |
|---|
| Model | Distributed commit log | Traditional message broker (AMQP) |
| Throughput | Very high (millions/sec) | Moderate (tens of thousands/sec) |
| Message retention | Configurable (days/forever) | Deleted after ack |
| Replay | Yes — re-read from any offset | No — once consumed, gone |
| Routing | Topic + partition key | Rich (exchanges, bindings, routing keys) |
| Best for | Event streaming, analytics, log pipelines | Task queues, RPC, complex routing |
How do you guarantee message ordering in Kafka?
Ordering is guaranteed only within a single partition. To preserve order for related events (e.g. all events for order #123):
- Use the same partition key (order ID) — hash routes to the same partition.
- Use a single partition topic if global ordering is required (limits throughput).
- Don't increase partitions without understanding the ordering impact.
producer.send(new ProducerRecord<>("orders", order.getId(), event));
// all events for same order ID land in same partition
How does Kafka retain messages?
Messages are stored on disk (sequential writes — very fast). Retention controlled by:
- retention.ms — time-based (default 7 days).
- retention.bytes — size-based per partition.
- log compaction — keeps only the latest value per key (like a changelog/topic).
Consumers track their own offset — slow consumers don't block others. But if a consumer falls behind beyond retention, data is lost for that consumer.
What is a consumer group rebalance?
When consumers join or leave a group (startup, shutdown, crash, timeout), Kafka reassigns partitions among remaining consumers. During rebalance, consumption pauses — a "stop-the-world" event.
Triggers: new consumer joins, consumer misses heartbeat (session.timeout.ms), partition count changes.
Mitigate: cooperative rebalancing (incremental), static group membership, keep max.poll.interval.ms above processing time.
What is the Schema Registry?
A separate service (Confluent) that stores schemas for Kafka message keys and values. Supports Avro, JSON Schema, and Protobuf.
Benefits:
- Schema evolution — backward/forward compatibility rules.
- Compact serialization — Avro stores schema ID in the message, not the full schema.
- Contract enforcement — producers can't publish incompatible data.
What is Kafka Connect?
A framework for streaming data between Kafka and external systems without writing custom producer/consumer code. Uses connectors:
- Source connectors — ingest from DB (Debezium CDC), files, APIs into Kafka topics.
- Sink connectors — export from Kafka to Elasticsearch, S3, JDBC, etc.
Runs as a distributed, scalable service with fault tolerance and offset management built in.
What is Kafka Streams?
A client library (not a separate cluster) for building real-time stream processing applications. Provides:
- map, filter, groupBy, aggregate, join operations on streams.
- Stateful processing with local RocksDB stores.
- Exactly-once semantics within the Kafka ecosystem.
- Windowing (tumbling, hopping, session windows).
Alternative: Apache Flink or ksqlDB for more complex processing at scale.
StreamsBuilder builder = new StreamsBuilder();
builder.stream("orders")
.groupByKey()
.count()
.toStream()
.to("order-counts");
KRaft vs ZooKeeper — what changed?
Historically, Kafka depended on Apache ZooKeeper for cluster metadata, leader election, and controller management. KRaft (Kafka Raft) mode removes this dependency — Kafka manages its own metadata quorum using the Raft consensus protocol.
Benefits: simpler deployment (one less system), faster metadata operations, better scalability for large partition counts. KRaft is the default in Kafka 3.3+.
Key Kafka metrics to monitor.
- Consumer lag — how far behind consumers are. High lag = processing bottleneck or consumer failure.
- Under-replicated partitions — replicas falling behind leader. Risk of data loss.
- Request rate / latency — broker throughput and response times.
- Active controller count — should always be 1.
- Offline partitions — partitions with no leader. Critical alert.
Tools: Prometheus + Grafana, Confluent Control Center, LinkedIn Burrow for lag monitoring.
How do you handle poison messages in Kafka?
A poison message repeatedly fails processing and blocks the partition. Patterns:
- Dead Letter Topic (DLT) — after N retries, publish the message to a separate topic for manual inspection.
- Skip and commit — log the error, commit offset, move on (data loss risk).
- Quarantine — store in a DB with error details for replay after fix.
Spring Kafka and Kafka Connect have built-in DLT support.
Using Kafka with Spring Boot.
spring-kafka provides:
@KafkaListener(topics = "orders", groupId = "processor") — declarative consumer.KafkaTemplate — send messages programmatically.- Auto-configuration from
spring.kafka.* properties. - Error handling with
DefaultErrorHandler and dead letter topics. @RetryableTopic — automatic retry with backoff to DLT.
@KafkaListener(topics = "orders", groupId = "processor")
public void handle(Order order) {
orderService.process(order);
}
What are compacted topics?
Log compaction retains only the latest value for each key — older values with the same key are eventually removed. The topic acts as a changelog (key → latest state).
Use cases: user profile updates, config changes, CDC snapshots. Combined with retention.ms for time + key-based cleanup.
How do you choose the number of partitions?
Consider:
- Target throughput — more partitions = more parallel consumers and producers.
- Consumer parallelism — max consumers per group = partition count.
- Ordering needs — fewer partitions = easier ordering.
- Broker resources — each partition has overhead (files, memory).
Rule of thumb: start with # of consumers you need × 2. Increasing partitions later is possible but can break key-based ordering if keys hash differently.
What is an idempotent producer?
With enable.idempotence=true, the producer assigns a PID (Producer ID) and sequence numbers per partition. If a batch is retried due to network error, the broker deduplicates using the sequence number — preventing duplicate messages within a session.
Required settings: acks=all, retries > 0, max.in.flight.requests.per.connection ≤ 5.
props.put("enable.idempotence", true);
How is Kafka used in event sourcing?
Event sourcing — store state changes as a sequence of events, not current state. Kafka's durable log is a natural event store:
- Commands produce events to a topic.
- Events are the source of truth.
- Current state is derived by replaying events (or reading from a materialised view).
CQRS — separate write (command) and read (query) models, connected via Kafka events.
