Best Practices and Interview Q&A

Operational Best Practices

• Design partition counts for broker count and load headroom.
• Use acks=all and sets replication factor 3 on production.
• Keep keys for ordering; avoid unbounded key cardinality.
• Monitor consumer lag and broker disk usage.
• Enable compression on the wire for throughput.

Design Best Practices

• Treat topics as contracts with versioned schemas.
• Use consumer groups for independent consumers.
• Compaction for stateful topics, retention for events.
• Test rebalancing behavior before adding consumers.

Interview Q&A

Q: How does Kafka guarantee ordering?

A: Per partition, records are appended in order and offsets commit sequentially. Keys route same-key records to the same partition.

Q: Kafka vs a traditional queue?

A: Kafka is a replayable, multi-consumer log with durability rather than a single-delivery queue, letting several groups read independently at different offsets.

Q: Why limit a consumer group to the partition count?

A: Each partition feeds one consumer at a time, so extra members sit idle while those partitions are held.

Q: How do you prevent duplicate event processing?

A: Use idempotent producers with transactions, consume with read_committed, and store processed record IDs to deduplicate on the application side.

Key Points

• Ordering, durability and schemas drive design.
• Lag, offsets and disk are the metrics to watch.
• Groups scale consumption to partition count.
• Exactly-once is reachable but costs complexity.