The Kafka Streams documentation divides into four layers: the Javadoc API reference, the configuration reference for streams, producer and consumer parameters, the architecture pages covering state stores and fault tolerance, and the upgrade guide for version-to-version migration.
The API layer centres on the KafkaStreams client and its interfaces, and knowing which layer answers which production question is most of the skill of using them. The library itself is covered on the Kafka Streams page; this one is the map.
Core needs for production
Two of those layers earn special attention in production. The configuration reference matters because Kafka Streams sits on top of a producer and a consumer, so its tuning surface is the union of three parameter sets, with streams-specific settings such as replication.factor for internal topics and commit.interval.ms layered over the client configs. That means the producer and consumer configuration pages are part of the Streams tuning surface too, not separate reading. And the state store pages matter because stateful operations persist to local RocksDB stores backed by changelog topics, which is the machinery behind every fault-tolerance and recovery question. Those changelog topics are ordinary Kafka topics with compaction turned on, so their retention and sizing behave exactly as they do anywhere else.
Where the official pages run thin is visibility: the documentation describes how streams applications work, and it does not give you a way to see whether yours is working. Monitoring stream processing state, tuning performance and confirming application reliability is tooling territory rather than documentation territory.
Troubleshooting and operations
The error-handling documentation centres on two interfaces, and they split the failure surface cleanly. Kafka Streams routes failed records through DeserializationExceptionHandler, for records that cannot be read on the way in, and ProductionExceptionHandler, for records that cannot be written on the way out. Everything else lands in the uncaught exception handler, which decides whether a stream thread dies, replaces itself, or shuts the application down. The dead-letter wiring around these interfaces is covered in the DLQ guide.
What the documentation does not cover is the state a Streams application gets stuck in afterwards, and we carry the same gap in our own product: our backlog holds a ticket to make the console more resilient by identifying the streams/ERROR state and restarting all streaming compute, and even that requires modifying the idempotence of our composite system. If surviving a stream thread that dies and does not come back is something you plan for, the uncaught exception handler is the documented half and the restart machinery is the half you build.
Metrics are exposed over JMX at several granularities: streams state, stream-thread, task, state store and RocksDB, covering lag, processing rate and thread states. The documented metric names are the raw material, and the operational questions they answer are which topology is running, which tasks are assigned where, and whether state stores are healthy. That task-level view is genuinely difficult to get at without dedicated tooling, which is why rebalance behaviour is where documentation reading turns into debugging: even experienced teams hit rebalances after which an instance or its state store does not recover fully, and the docs’ threading model pages are what you reason from.
Scaling rules come straight from the task model. A topology divides into tasks by partition, tasks are assigned across instances and threads, and an instance beyond the task count adds nothing. The documentation’s threading and capacity pages state the rule, and the practical corollary is that scaling a Streams application means scaling partitions, threads and instances together rather than any one of them alone.
The reason I tell people to read the metrics pages before they need them: standard Kafka metrics signal that something is wrong and rarely explain the root cause. If I have 100% CPU or 100% network throughput, standard metrics will tell me something is wrong, and they will not tell me why. The documented Streams metric names are the vocabulary of the investigation you will eventually run, and learning them during an incident is the expensive way. Know before the bad day which metric distinguishes a stalled thread from a rebalancing task from a state store restore in progress, because from the outside all three look like “lag going up”.
The upgrade guide is the layer that hides the most value. Derek, our co-founder, flagged one example to our own team: Kafka Streams introduced a new state store type, versioned key-value stores, for storing multiple record versions per key, which enables timestamped retrieval operations to return the latest record per key as of a specified timestamp, with KIP-889 and KIP-914 carrying the detail and improved processing semantics applying when the new store type is used in the DSL. That is exactly the class of change that sits in the upgrade guide and never reaches the people who would benefit from it.
One documentation detail that has saved real designs: since Kafka 2.6, KStream#repartition() (KIP-221) gives you an explicit repartition step when source topics cannot be co-partitioned. Joins in Kafka Streams require co-partitioned inputs, and before that operator the workaround was manual through-topics that half the team did not know existed. It is exactly the kind of fix that sits documented and unread while people rebuild it by hand.
FAQ
How does Kafka Streams handle bad records?
Through two documented interfaces that split the failure surface cleanly: DeserializationExceptionHandler for records that cannot be read on the way in, and ProductionExceptionHandler for records that cannot be written on the way out. Everything else lands in the uncaught exception handler, which decides whether a stream thread dies, replaces itself, or shuts the application down.
How do I scale a Kafka Streams application?
By the task model: a topology divides into tasks by partition, tasks are assigned across instances and threads, and an instance beyond the task count adds nothing. Scaling a Streams application means scaling partitions, threads and instances together rather than any one of them alone.