KRaft is Kafka’s built-in consensus protocol, and it removes ZooKeeper from the architecture entirely. Cluster metadata moves into an internal Kafka topic, __cluster_metadata, managed by a quorum of controller nodes running the Raft protocol, which leaves you one system to deploy, secure and monitor where there used to be two.
That second system was the painful one. Running ZooKeeper well was a genuinely horrible operational chore, and because it was, organisations cut corners: the classic mistake was running one under-provisioned ZooKeeper ensemble shared across many Kafka clusters, where the more traffic you put on it, the greater the odds a single ZooKeeper issue takes every one of those clusters down at once. The first page of the old Kafka operations playbook was “deploy a dedicated ZooKeeper cluster for every Kafka cluster” for exactly that reason. KRaft makes the whole problem disappear, and that alone makes operations significantly easier.
The timeline matters because it sets your upgrade path. KRaft became available in Kafka 3.3, reached full feature parity with ZooKeeper mode in Kafka 3.9, and is mandatory from Kafka 4.0. A 3.x cluster still on ZooKeeper must migrate to KRaft before it can upgrade to 4.x. This page covers the migration, the limits, the configuration and the day-2 operations in that order. The wider context sits in the Kafka architecture guide and the complete Kafka guide.
Migration guides and cluster sizing
Migration is no longer optional. Kafka 3.x clusters running with ZooKeeper must complete the KRaft migration before upgrading to 4.x, so the question is when and how, not whether. The migration replaces ZooKeeper’s role with the controller quorum and moves all cluster metadata into the event-sourced __cluster_metadata log.
For a greenfield cluster there is no decision to make: you deploy KRaft. The recommended baseline is Kafka 4.0 or later, KRaft only, in isolated mode: three dedicated controller nodes plus three or more brokers, spread across three availability zones. The controller quorum stays at three nodes for most clusters, five where higher availability is required.
Monitoring changes with the migration. ZooKeeper-era control plane metrics are replaced by KRaft-native equivalents, including consensus metrics under kafka.server:type=raft-metrics and a new family of metadata loader and controller queue metrics. If the migration plan does not update dashboards and alerts along with the brokers, you end up with a cluster your monitoring can no longer see into.
The migration itself is well trodden. The live ZooKeeper-to-KRaft migration path has been GA since Kafka 3.6, September 2023, and the largest known migration is already done: Confluent Cloud moved thousands of clusters to KRaft without breaching SLAs.
The risk sits in the tooling around your cluster instead. With Kafka 4.x in KRaft mode, one widely used open-source UI was still showing inconsistent partition leaders on every page refresh, because it queried brokers instead of the KRaft quorum controller. That is the migration lesson in one bug: audit everything that reads cluster metadata, dashboards, UIs, automation scripts, before you flip, because tools written in the ZooKeeper era make ZooKeeper-era assumptions.
The audit is more tractable than it sounds, because the KRaft API endpoints are quite small. When Tom Crowley, our founding engineer, added KRaft support on our side, the surface came down to listing the replicas in the quorum and unregistering a broker from a Kafka cluster. A handful of endpoints is all that separates a tool that speaks KRaft from one that does not, and a tool that has not crossed that line will simply show you nothing rather than tell you it cannot.
Production stability and limits
KRaft reached full feature parity with ZooKeeper mode in Kafka 3.9, which removed the last practical impediment to migration for the early-adopter holdouts.
The scale story is the reason KRaft exists. The ZooKeeper architecture imposed a hard ceiling: metadata handling degraded substantially as partition counts grew past roughly 200,000 per cluster, with the old rules of thumb at 4,000 partitions per broker. KRaft’s consolidated metadata management removes that bottleneck. Lab tests have demonstrated stable operation at 2 million partitions per cluster, and real-world production clusters typically run in the hundreds of thousands.
The 2-million-partition number is what the metadata layer can survive in a lab. It is not what your brokers can serve in production, where disk I/O and network I/O still have physical limits. The per-broker economics of partitions did not change, which is why a practical bound of 10,000 to 20,000 partitions per broker remains sensible for most workloads even on KRaft: partition count still costs memory, file handles and recovery time on every broker regardless of where the metadata lives.
Parts of the wider ecosystem are still catching up here too. Some monitoring and management tools were built in the ZooKeeper era and do not yet fully support KRaft, and the tools are where migrations actually stall, so before you plan a migration, verify that everything in your stack that reads cluster metadata works against a KRaft cluster.
Configuration and architecture
The first configuration decision is node roles. Every KRaft node declares process.roles as broker, controller, or both. Combined mode is fine for development stacks, and production runs isolated roles: an odd number of dedicated controller nodes, separate from the brokers, so the consensus layer is insulated from broker-side I/O pressure and garbage collection pauses.
The controller quorum has hard requirements. A production deployment needs a minimum of three controller nodes for quorum, five for higher availability, on dedicated nodes with fast SSD storage, spread across separate availability zones, with the controller listener isolated from client listeners. The quorum is defined in controller.quorum.voters, listing each controller’s node.id and address, and every node’s identity is fixed by its node.id.
Under the quorum sits an event-sourced metadata log, and it is what makes the scale claims real: Raft-based metadata management supports millions of partitions. Its health is directly observable, with MetadataErrorCount (kafka.controller:type=KafkaController,name=MetadataErrorCount) counting failed metadata log operations, which should be zero.
On broker sizing around the quorum, my advice runs against the instinct to buy the biggest boxes: avoid the largest available nodes. Mid to large brokers leave you headroom to scale vertically when you need capacity fast, and once you are on the largest instance the only move left is horizontal scaling, which on Kafka means partition movement at the worst possible moment. The broker side of that equation, from server.properties through safe restarts, is covered in the Kafka broker guide.
And a configuration discipline that KRaft makes easier to follow: after any rollback, diff the running configuration against your baseline, and audit it against documented best practice periodically rather than after incidents. Config drift is the quiet failure mode of long-lived clusters, a rollback that half-applied, a break-glass override somebody forgot to remove, and the whole class of problem is invisible unless something is comparing running state to intended state. The metadata log gives you a single place where cluster configuration lives. Use it.
Operations and day-2 management
Day-2 KRaft operations start with watching the quorum. Quorum health metrics from kafka.server:type=raft-metrics belong in monitoring from day one, alongside the KRaft-specific MBeans that appear with the mode: FencedBrokerCount and LastAppliedRecordLagMs, the second of which tells you how far a node has fallen behind the metadata log. The ZooKeeper-related MBeans they replace are simply gone, so old dashboards silently show nothing.
The cluster-level alert set does not change. Under-replicated partitions, ActiveControllerCount and OfflinePartitionsCount remain the most impactful signals, with ActiveControllerCount now describing the KRaft controller: since Kafka 4.0 it is the KRaft controller that manages cluster metadata and leader election, and exactly one controller must be active.
Security gets simpler, and that is a genuine operational win. KRaft provides a single security model, removing the separate SASL and ACL configuration that ZooKeeper required, which eliminates a second attack surface. The remaining work is securing the inter-controller channel with TLS and SASL like any other listener, instead of maintaining a parallel ZooKeeper security regime.
The day-2 point I push hardest: managed Kafka health is a shared responsibility. Managed service providers often do a great job of monitoring your cluster, but there are still metrics you want to keep an eye on yourself, and the quorum is the clearest case: a metadata-apply-error-count above zero means metadata corruption on the broker, and your own monitoring is what will catch it.
For a sense of what disciplined day-2 operations look like at the far end, JPMorgan Chase runs more than 100 Kafka clusters and dealt with the operational burden by building a health index, a centralised control plane, and an orchestrated patching pipeline: monitoring, patching and quota enforcement, automated. Most Kafka topologies need a fraction of that machinery, and every one needs some deliberate version of it, and KRaft simplifying the architecture is precisely the moment to build it, while the pieces are one system instead of two.
FAQ
Does Kafka still need ZooKeeper?
Not on any current version. Kafka 4.0 and later run KRaft only, with cluster metadata managed by a quorum of controllers, while versions earlier than 4.0 rely on ZooKeeper unless they have migrated. A 3.x cluster still on ZooKeeper must complete the KRaft migration before it can upgrade to 4.x.
When did KRaft become production-ready?
KRaft became available in Kafka 3.3, but full feature parity with ZooKeeper mode only arrived in Kafka 3.9. The live ZooKeeper-to-KRaft migration path has been GA since Kafka 3.6 in September 2023, and KRaft is mandatory from Kafka 4.0.