A Kafka broker is a single Kafka server that stores partition data on disk and serves produce and fetch requests from clients. Brokers form a cluster, replicate partitions between each other, and elect new partition leaders when one fails.
I am a Solutions Architect at Factor House, and most of what follows comes from broker incidents I have either worked or watched closely. This page covers configuration, troubleshooting, monitoring, maintenance, metadata and network tuning, in that order. The wider picture sits in the Kafka architecture guide and the complete Kafka guide.
Configuration parameters
Every broker reads its identity and behaviour from server.properties at startup, and the Apache Kafka broker configuration reference documents every parameter below. Identity comes first: broker.id names the broker uniquely within the cluster, node.id does the same job on KRaft clusters, and log.dirs lists the disks where partition data lives.
The listener pair is the most misconfigured entry in the file. listeners is the address the broker binds. advertised.listeners is the address the broker hands back to clients in metadata responses, and it is the one clients actually connect to. When a client reaches the bootstrap address fine and then times out, the advertised address is almost always the problem, because the broker returned a hostname the client cannot resolve or route to.
Retention is the other block worth knowing cold. log.retention.hours or log.retention.ms bounds retention by time, log.retention.bytes bounds it per partition by size, and log.segment.bytes sets the segment size that retention deletes in units of. Retention applies per topic partition, and a topic-level override always beats the broker default.
The reason I treat broker configuration as an operational surface rather than a set-and-forget file is an incident of ours. An MSK cluster upgrade of ours failed and rolled back, and the provisioned storage throughput reverted while the replica fetcher and IO thread counts we had raised did not, so the config drift ran silently for six months. When the cluster eventually came under recovery load, the slow provisioned storage could not handle the extra throughput those two raised settings were driving, and the whole cluster went unstable. You could not even tell it was broken from where it hurt: the first visible symptom was rising producer error counts, two steps removed from the disk that was actually the problem. The lesson we took from it: after any rollback, diff the running configuration against your baseline. The cluster management guide covers the wider discipline.
Troubleshooting and error codes
Connection drops usually trace to one of three places. The advertised listener returning an unreachable address, an idle connection closed by the broker once connections.max.idle.ms expires after ten minutes of silence, or a proxy in front of the brokers with a shorter idle timeout than the clients expect. Kafka clients need to reach a specific broker, so the network infrastructure in front of a cluster is routers and proxies rather than load balancers, and any of it can quietly kill connections. A producer retries through these invisibly until it cannot, which is why the first symptom is often a latency spike rather than an error.
Out-of-sync replicas are a health signal, not an error. A follower that has not caught up to the leader within replica.lag.time.max.ms, 30 seconds by default, is dropped from the in-sync replica set, and the partition shows as under-replicated. Brief ISR shrink during restarts is normal. Sustained under-replication points at a broker that cannot keep up, a saturated disk, or a network path problem.
With unclean.leader.election.enable at its default of false, a partition whose in-sync replicas are all gone goes offline rather than electing a stale follower. Setting it true restores availability at the price of acknowledged writes. If you flip it during an incident, write down what you lost.
Crash loops nearly always start on disk. A full volume in log.dirs kills the broker, restart triggers log recovery, recovery fails or the disk fills again, and the loop continues. On a managed service this gets worse, because sometimes you cannot add more disk: hit the storage maximum on an MSK broker and there is no capacity left to grant, and recovering a cluster from that state is genuinely hard. Watch disk space closely enough that you never find out. The other frequent cause is a heap misconfiguration that turns startup log recovery into an OutOfMemoryError. Read the first fatal log line, not the most recent one.
Metrics and monitoring
Four JMX metrics, all from Apache Kafka’s own monitoring documentation, cover most broker incidents.
Under-replicated partitions should be zero: kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions
Exactly one active controller must exist across the whole cluster: kafka.controller:type=KafkaController,name=ActiveControllerCount
Offline partitions must be zero: kafka.controller:type=KafkaController,name=OfflinePartitionsCount
And a growing request queue means requests are arriving faster than the IO threads can clear them: kafka.network:type=RequestChannel,name=RequestQueueSize
Behind those sit the saturation signals. The IO thread pool is running out of headroom when average handler idle drops below about 0.3: kafka.server:type=KafkaRequestHandlerPool,name=RequestHandlerAvgIdlePercent
Request latency lives in kafka.network:type=RequestMetrics,name=TotalTimeMs with request=Produce or request=Fetch, and its queue-time component tells you whether the time went to waiting or to work. Throughput is BytesInPerSec and BytesOutPerSec under kafka.server:type=BrokerTopicMetrics. CPU and disk utilisation come from the host, not from Kafka, and most teams scrape all of it into Prometheus with the JMX exporter.
Per-metric thresholds and collection setups are in the broker monitoring guide, and the monitoring tools comparison covers what to run on top.
Cluster maintenance
A rolling restart is safe when it is boring. Restart one broker at a time, let controlled shutdown migrate partition leadership away first, and wait for under-replicated partitions to return to zero before touching the next broker. Skipping the wait is how a routine restart becomes an outage: in a cluster with well-distributed data, taking a second broker down is close to a guarantee that some partition loses its last in-sync replica. Anything short of the wait is operating on luck, and luck is not a strategy.
Your tooling has to treat restarts as routine too. Our co-founder and CEO Derek Troy-West made Kpow’s AWS integration more lenient with rolling restarts, because a broker that disappears for ninety seconds during a planned roll is not an incident, and tooling that pages on every planned restart trains people to ignore the page that matters.
Decommissioning runs in the opposite order. Move every partition off the broker first with kafka-reassign-partitions.sh, confirm it holds no replicas, then shut it down and deregister it so the cluster stops expecting it back. Replacing a dead broker is the same mechanics reversed: bring up a new node with the dead broker’s id and the cluster re-replicates its partitions onto it.
Metadata and state
On disk, a broker is a directory tree of partition logs. Each partition is a set of segment files, a .log file of records plus .index and .timeindex files mapping offsets and timestamps to file positions. Writes append to the active segment only, older segments are immutable until retention deletes them, and reads are served through the operating system page cache, which is why brokers want memory well beyond the JVM heap. Tiered storage follows the same rule: a segment is only copied out to object storage like S3 once it is closed, so a segment that takes a long time to close keeps occupying local disk regardless of what your tiering policy says.
Cluster state lives elsewhere. On a modern cluster the broker registers with the KRaft controller quorum and follows the internal __cluster_metadata topic to learn topic configurations, partition assignments and leadership. Older clusters did the same through ZooKeeper. The KRaft guide covers the quorum, the migration and its day-2 operations. Consumer progress is cluster state too, stored on the brokers in the internal __consumer_offsets topic and covered in the offsets guide.
Network tuning
The knobs stack in layers. At the socket layer, socket.send.buffer.bytes and socket.receive.buffer.bytes size the TCP buffers, which matters most on high-latency links between sites. socket.request.max.bytes caps a single request at 100 MB by default, and message.max.bytes caps a record batch at roughly 1 MB, a limit producers and consumers have to agree with.
Above the sockets sit the thread pools. num.network.threads moves bytes on and off the wire, and num.io.threads does the disk work. In my talk on Kafka operational issues I put it plainly: the replica fetcher count controls how many threads exist to fetch data, the IO thread count controls the IO thread pool, and both settings move disk and network performance together. They were also the two settings whose drift caused the incident above, so tune them with the disk they will hit in mind, not just the CPU in front of it.
Quotas are the guard rail on all of it. On a cluster with a lot of topics and partitions, a new consumer running a large backfill can take enough network bandwidth to degrade your regular production workloads, and quotas are what stop one client from queueing everyone else’s requests behind its own. Set them before that happens, not after.
FAQ
What does a Kafka broker do?
A Kafka broker stores partition data on disk as segment files, serves produce and fetch requests from clients, replicates partitions to other brokers, and follows cluster metadata from the KRaft controller quorum, or from ZooKeeper on older versions.
Why can clients reach the bootstrap server but not the broker?
Because advertised.listeners is the address the broker hands back to clients in metadata responses, and it is the one clients actually connect to. When the broker advertises a hostname the client cannot resolve or route to, the bootstrap connection works and everything after it times out.
How do I restart a Kafka broker safely?
Restart one broker at a time, let controlled shutdown migrate partition leadership away first, and wait for under-replicated partitions to return to zero before touching the next broker. Skipping the wait is how a routine restart becomes an outage.
What is an under-replicated partition?
A partition where a follower has not caught up to the leader within replica.lag.time.max.ms, 30 seconds by default, and has been dropped from the in-sync replica set. Brief shrink during restarts is normal, and sustained under-replication points at a broker, disk or network problem.