A production Kafka tutorial covers what beginner guides skip: how to change a running cluster without downtime, which signals to trust when something breaks, and which client settings decide whether messages survive, and that is what this page covers.
For the concepts themselves, start with what Apache Kafka is and come back.
Production and scale details
A production Kafka cluster has to change while it is running. Configuration changes and version upgrades are applied with a rolling restart: brokers restart one at a time while partition leadership moves to in-sync replicas on the remaining brokers, so producers and consumers stay connected throughout. An upgrade that requires stopping the whole cluster is a design failure, not a maintenance window.
A failed upgrade is where that gets interesting, because a rollback is not done until every change has been verified reverted. On one MSK cluster of ours the upgrade failed mid-process and rolled back automatically, and the provisioned storage throughput reverted with it while the increased replica-fetcher and IO-thread counts did not. That drift ran for six months without a single symptom. Then one disk failed, the misconfigured replica fetching hammered it during recovery, and producers across multiple teams started seeing timeout exceptions.
The habits that catch that are cheap. Treat broker config changes like code where you can, version-controlled and applied through CI. After any rollback, diff the running config against your baseline and confirm it is actually okay, and periodically audit the running config against your documented best practices. And when you are resolving an incident, stop and make one change at a time, rather than rolling a settings change and a disk increase into the same window. My own team used to do exactly that.
Scaling a traditional Kafka cluster is not instant. Adding brokers triggers partition reassignment, which physically copies data between broker disks over the network, and that movement competes with production traffic for bandwidth. Capacity in practice is set by partition count, consumer group design and broker sizing well before it is set by hardware.
Broker tuning operates at three layers. The JVM layer is garbage collection: long GC pauses cause brokers to miss heartbeats and drop out of the cluster. At the operating system layer, file descriptor limits and the page cache do the damage: Kafka holds a file handle per log segment, and a broker at the default OS limit stops accepting connections. On the network, socket buffer sizes cap throughput on high-latency links between brokers and clients.
Production security is layered, not a single control. A hardened deployment isolates the network with VPC isolation, security groups and private listeners, encrypts all traffic with TLS, authenticates clients with SASL or mutual TLS, and authorizes every principal with ACLs or an external authorizer. A multi-tenant cluster additionally needs role-based authorization at the topic, consumer group and connector level, audit logging, and integration with an enterprise identity provider over SAML, OIDC or LDAP.
Most of what pages like this list as “advanced configuration” is really the same discipline applied in different places. Most Kafka problems are not Kafka bugs. They are misconfigurations, missing observability, or reasonable decisions made without full context, and the production topics in this section, upgrades, tuning, security, are where those reasonable decisions get made. The quick-win checks I give teams are unglamorous for exactly that reason: offset reset defaults, poll interval limits, DLQ loops, retention sizing, message size limits, linger.ms, client library choice, transactional producer usage. None of them is an architecture change, and every one has caused a real incident somewhere.
The security layer has its own version of this. The practical failure mode in Kafka authorization is not a breach, it is permission creep: ACLs accumulate over time through broad grants and principals nobody ever revokes, until the ACL list describes the org chart of three years ago. Audit it like you audit retention.
Troubleshooting and operations
Kafka troubleshooting starts from a small set of signals. The four most impactful are under-replicated partitions, ActiveControllerCount, OfflinePartitionsCount, and the consumer lag trend. Broker CPU, memory and network throughput tell you that a problem exists, and rarely tell you what caused it.
An under-replicated partition is a partition whose follower replicas have fallen behind the leader and dropped out of the in-sync replica set. The usual causes are a failing broker, saturated disk I/O, or network congestion between brokers. UnderReplicatedPartitions above zero for more than five minutes is a critical alert condition, because writes to affected partitions are one broker failure away from data loss.
Consumer lag is the distance, measured in offsets, between the newest record in a partition and the position a consumer group has committed. Lag has to be tracked at partition granularity, and reading it right is its own discipline: a group-level total hides one stuck partition behind healthy ones, and the stuck partition is usually the story. Clearing lag safely means first distinguishing a slow consumer, which needs scaling or tuning, from a stuck consumer, which is blocked on a poison message or a rebalance loop and does not improve with more instances.
A dead broker is survivable by design. Every partition it led elects a new leader from the in-sync replicas on other brokers, and clients rediscover leadership automatically. Replacing the broker means bringing up a new one and letting replication rebuild its data, either by reusing the dead broker’s id or by reassigning its partitions. No data is lost provided the replication factor was at least three and producers wrote with acks=all.
Of everything on this page, disk is the one I want tattooed somewhere. Running out of disk is about the worst state a Kafka cluster can be in, and it is very hard to recover from: you cannot write, you often cannot cleanly delete, and every recovery action needs the resource you have run out of. Retention sizing is not housekeeping, it is the thing standing between you and that state.
Retention set too high on a high-throughput topic is the slow-burn version. Seven days is a common default, and gigabytes per hour for seven days adds up to a lot of disk, so the volume fills over weeks and then is suddenly urgent. The fix is unglamorous: monitor disk usage per topic, and set retention.bytes as well as retention.ms on high-throughput topics so a topic’s on-disk size is capped even if that means deleting data earlier than the time-based retention would. You would not believe how many setups I have seen where disk usage is not monitored at all.
One habit belongs on that signal list even though it is not a metric: do not wait until you are in a crisis to open a support case. Several of the incidents I have worked were only cracked open by a vendor pointing out something the team could not see from the inside, and the MSK consumer group incident was diagnosed exactly that way, by a support ticket, after our own dashboards had stayed green throughout. Support is a debugging tool, not a last resort.
The published incident record backs the severity. One post-incident analysis of a Kubernetes-hosted Kafka failure documents a cascading broker crash that pushed consumer lag to 14 hours across 10,000 topics. And the failure modes are not always inside Kafka: PagerDuty’s staging Kafka hosts went intermittently unresponsive for tens of seconds at a time, producing client connectivity failures, under-replicated partitions and leader elections, with the root cause outside the brokers entirely. When the signals fire together like that, resist the reflex to fix Kafka first. Find what the host is doing.
Architecture and code integration
Client configuration decides the delivery guarantee, and the defaults favour throughput over safety. Setting acks=all makes the partition leader wait for every in-sync replica to confirm a write before acknowledging it. Combined with retries and idempotence enabled on the producer, this prevents both message loss and duplication through transient failures. Production producers set these values explicitly rather than trusting defaults.
A schema registry manages the contract between producers and consumers. Every message schema is versioned in one central place and checked for compatibility before a producer can publish a change, so a field removal or type change that would break downstream consumers is rejected at build or publish time instead of discovered in production. In a microservices architecture this is what allows teams to evolve their events independently.
Modern Kafka clusters run in KRaft mode, which replaces ZooKeeper with a Raft quorum built into Kafka itself. The result is one system to deploy, secure and monitor instead of two, and faster controller failover on large clusters.
Prometheus and Grafana are the most common self-hosted monitoring stack for Kafka. Broker-side alerts cover under-replicated partitions, active controller count and offline partitions. On the producer side, the alerts that matter are delivery failures, back pressure, queue delay and retry rate, each mapping to a distinct failure mode in the client rather than the cluster.
I gave a talk built entirely on this section’s theme: four real production incidents, every one of them survivable with the configuration and monitoring described above, none of them survived gracefully at the time. The pattern across all four was identical. A reasonable decision made without full context, a missing or misread signal, and then a response, usually scaling Kafka itself, that made the underlying problem worse before the real cause surfaced.
That is why my advice on this section is to treat alert rules as code you ship, not dashboards you admire. Concrete thresholds beat vibes: a producer retry-rate alert firing when the five-minute rate exceeds 10 for three minutes tells you something specific is wrong at the client, before the broker-side graphs move at all. The delivery numbers achievable when the whole chain is configured deliberately are real: DoorDash holds data loss under 0.001% in async mode, at billions of messages a day. Not because Kafka is magic. Because every one of the settings on this page was chosen on purpose.
FAQ
How hard is Kafka to learn?
Kafka’s concepts take an afternoon. A topic is an append-only log, a partition is its unit of parallelism and ordering, producers write, and consumer groups share the reading. What takes longer is operating it: the failure modes, the configuration interactions and the recovery procedures this page covers. The learning curve is operational, not conceptual, which is why a production tutorial spends its time on upgrades, troubleshooting and monitoring rather than on the API.
Is Kafka a part of DevOps?
Kafka is infrastructure, not a DevOps tool. It is typically owned by a platform or data engineering team and operated with the same discipline DevOps applies elsewhere: configuration under version control, changes through CI, monitoring from day one. Application teams own their producers and consumers; the cluster itself is a shared platform with an operations rota.