Apache Kafka is an open-source distributed event streaming platform that stores records in ordered, partitioned, replayable logs called topics. Producers append records to a topic, consumers read them at their own pace, and the log persists whether or not anyone has read it. Where a traditional message queue deletes each message once it has been delivered, Kafka keeps the record in the log, and that retention is what everything else here builds on.
It was built at LinkedIn and given to the Apache Software Foundation, and it is now the default way large systems move events between services. If you want to understand the thinking Kafka came out of, the article Jay Kreps wrote at LinkedIn is still the best primer there is. It changed the course of my career, and when I met Jay last year I told him exactly that. I consider it required reading for anyone dipping their toe into Kafka.
I am a Solutions Architect at Factor House. Before that I ran Kafka in production at Block, Square and Cash App, spent four years building a distributed systems startup, and over more than fifteen years of working with Kafka I have consulted with around fifty companies on running it at every scale. Most of the caveats below come from watching them bite.
Core concepts
Kafka has five moving parts, and the relationships between them explain almost every operational surprise you will hit later.
A topic is a named, append-only log. It is not a queue and it is not a table.
A partition is a slice of a topic, and it is the unit that actually matters. Ordering is guaranteed within a partition and never across a whole topic. Each partition is assigned to exactly one consumer in a group at a time, which is the constraint that governs how far a consumer group can usefully scale.
A broker is a server holding partitions. Every partition has a leader and zero or more followers, and the leader handles all produce and fetch requests for that partition while the followers fetch from it to stay in sync.
A producer appends records. A consumer reads them, tracking its own position by offset rather than having the broker track it.
A special broker called the group coordinator manages consumer group membership and assignments. It is invisible until it is the thing that is broken.
The consequence worth internalising early: because a partition is assigned to exactly one consumer at a time, your partition count is your parallelism ceiling. Uber’s engineering team wrote about hitting this directly: their payment-processing consumers could handle roughly one event per second per partition, so reaching 1,000 events per second meant 1,000 partitions on the topic.
That cuts both ways. Adding partitions is not free for consumers, and you need to know where your consumers will start reading from before you scale. I have watched a service consuming from hundreds of topics, each with about 10 partitions, scale out to hundreds of instances and get slower, because the extra instances had no partitions left to claim and simply added coordination load.
Why it is used
Kafka is shared infrastructure. It is designed as a multi-tenanted, highly scalable backbone for moving data across an organisation, rather than a point-to-point pipe between two services. In most architectures that adopt it, the cluster becomes the central message bus for every real-time event in the system.
The scale it reaches is genuinely large. Cloudflare has run Kafka in production since 2014, handling hundreds of billions of events per day for log processing and analytics. By August 2021 Uber’s deployment had 200,000 partitions and processed 12 million messages per second.
It is worth being equally clear about where it fits badly, because “what is Kafka” articles rarely are.
Kafka handles millions of small messages per second extremely well, and handles large messages poorly. Strict ordering guarantees, partition rigidity, rebalancing overhead and replication cost make it an expensive fit for high-volume machine-learning training pipelines where sub-second latency is not actually required. And in traditional Kafka, scaling a cluster out means slow, network-intensive physical data movement between broker disks, so growth is not instant.
Netflix learned the topology lesson the hard way. Their original single-cluster design let consumer fan-out degrade ingest performance, because producers and consumers were competing for the same broker resources. The fix, written up on their engineering blog, was a two-tier topology separating fronting clusters from consumer clusters.
My own rule, after enough incidents: scale a Kafka cluster only when you genuinely need to, for headroom or real throughput. Scaling is the first instinct when something is slow, and it is frequently the thing that makes it worse.
The honest summary is that operating Kafka at scale is harder than understanding Kafka. The concepts above take an afternoon. The failure modes take longer, which is why the rest of the complete guide to Kafka exists.