ksqlDB is the streaming SQL layer for Kafka: it lets you express stream processing as SQL over topics instead of writing Java or Scala against the Streams API. Its two core objects, streams and tables, mirror Kafka’s own duality.
CREATE STREAM defines an unbounded sequence of events over a topic, and CREATE TABLE defines the current state per key. A working example is two statements: CREATE TABLE GRADES (ID INT PRIMARY KEY, GRADE STRING, RANK INT) WITH (kafka_topic = 'test_topic', value_format = 'JSON', partitions = 4); followed by CREATE TABLE TOP_TEN_RANKS AS SELECT ID, RANK FROM GRADES WHERE RANK <= 10;. The second statement is a persistent query that runs continuously from that point on. Two statements over a topic named test_topic is genuinely the whole surface area of getting a persistent query running, which is both why ksqlDB is attractive and why people underestimate what that second statement actually starts.
Core documentation and syntax
Under the SQL surface, ksqlDB is fundamentally a REST API. The CLI and every client library are wrappers over REST calls to the server, even where the documentation presents it with the feel of a database driver, host and port configuration included. Knowing that changes how you debug it: any HTTP client can ask a ksqlDB server what it is running.
That database-driver framing is also where connections break. Tom Crowley, our founding engineer, points out that Confluent clearly want it to feel like a driver, giving you KSQLDB_HOST and KSQLDB_PORT rather than just a REST URL, so customers hand it a value with the scheme attached and get an unknown-host style error back. It is more or less the error psql has thrown for over twenty years for the same mistake. Before debugging anything deeper, check whether someone prefixed http:// onto the host.
One market fact belongs in any honest introduction. Confluent’s product positioning has shifted decisively toward Flink, and ksqlDB now receives minimal prominence in current Confluent documentation and engineering content. ksqlDB still works and still ships, and a team choosing it today should know its vendor’s centre of gravity has moved.
The conceptual trap with SQL-on-streams is the word SELECT. My colleague Falko Schwarz put it best from watching real users: most people using KSQL wanted a single “42” result, and instead received a stream they did not know how to handle. SQL trained everyone to expect a query to finish. In ksqlDB, a plain SELECT over a stream is a subscription, not an answer: it keeps emitting rows as events arrive, and the “result” never arrives because there is no end of the data. Pull queries against tables give you the database-shaped answer; push queries give you the stream. Teach that distinction before the syntax, because every early frustration with ksqlDB is someone expecting a pull and getting a push.
Production operations and scaling
ksqlDB runs in three deployment shapes: self-managed clusters you operate yourself, Confluent Cloud ksqlDB, and Confluent Platform ksqlDB. The operational difference is who owns the servers, because a ksqlDB cluster is a set of JVM server processes that scale by adding nodes, and every persistent query consumes CPU, memory and state on those nodes. Managed Kafka services in general handle cluster provisioning, scaling and operations, and the same trade applies at the ksqlDB layer: self-managing buys control and costs an operational surface.
Sizing follows the same logic as any JVM stream processor. Heavy workloads need deliberate heap sizing and JVM tuning, and the practical ceiling on a deployment is the state its queries hold rather than the query count. Monitoring runs over JMX like the rest of the Kafka ecosystem, with query timing metrics available in the performance settings surface, and the health question is per query, not per server: a healthy cluster can be running one degraded persistent query.
Whatever the deployment shape, the operational tooling should see all of it at once. Multi-cluster operations are the norm in real topologies, where one management surface covers a dozen clusters with their associated Connect, schema registry and ksqlDB deployments, because a query’s health only means something next to the topics and consumer groups it feeds.
The managed options do not remove the operations line item, and this is the point I make about every managed streaming service: health stays a shared responsibility, the argument the KRaft page makes in full. Managed ksqlDB runs the servers. It does not know whether your persistent query is producing correct output, whether its state has grown past what you budgeted, or whether the consumer group behind it is falling behind. Those need your monitoring and your configuration hygiene regardless of whose logo is on the console.
The half-running state is the one to respect. Our co-founder and CEO Derek Troy-West hit it while reading AWS startup logs for our own dev and demo environment: he noticed nullpointers to do with ksqlDB, raised a ticket, then realised at the end that it was probably just a ksqlDB instance in a half-running, bounced or down state, so not really anything to worry about. ksqlDB fails like a distributed system, not like a database process: partially, quietly, with symptoms in other components’ logs. Health checks should ask it questions over its REST API, not check whether the process exists.
Log lines are a poor substitute for that, and a customer report we worked through shows why. Tom had three theories for the error and only one of them was a bug in our code: a ksqlDB version bump breaking them, a recently-hosted Confluent Cloud ksqlDB being broken with the client, or user input error such as pointing it at an empty cluster. The error was firing inside the event loop of the ksqlDB client on an uncaught thread, and the logs were trimmed, so the context that would separate those three theories was the part that never made it into the file.
Integration and performance
ksqlDB reaches external systems through Kafka Connect. Sources bring database rows, logs and API events into topics where SQL can reach them, sinks carry query results out, and Connect itself keeps its configuration, offsets and status in dedicated Kafka topics, so the integration layer is stateful in exactly the way the query layer is. Operating the pair means watching connector states, where RUNNING, PAUSED, FAILED, UNASSIGNED and UNREACHABLE each mean a different failure and a different fix, and a stalled connector upstream presents as a quiet query downstream.
Persistent queries store their state the Kafka way. Every stateful query, a windowed aggregate, a join, a table, keeps its working state in internal Kafka topics, which is what makes a query fault tolerant: a restarted server rebuilds state from the changelog rather than losing it. The corollary is that query state is cluster data, and it costs storage and rebuild time like any other topic.
Troubleshooting an active pipeline works from three symptoms. Consumer lag on a query’s input topics means the query is falling behind its data. Processing delay with low lag points at the query itself, where query timing metrics show where the time goes. And out-of-memory errors are almost always state grown past the heap sizing, which is a capacity decision presenting as a crash. Debugging any of them is faster when the topology, internal state stores and per-query status are inspectable in one place, with consumer groups linked to the topologies and state-store partitions they serve.
When ksqlDB is the right tool, the payoff is fast and real. Reddit killed a two-hour instrumentation feedback lag by putting a ksqlDB-backed web application directly on the live Kafka pipeline, filtering events as they flowed, feedback in seconds. No new processing cluster, no Java service, SQL on the stream that already existed. That is the shape of a good ksqlDB use case: the data is already in Kafka, the question is expressible in SQL, and the answer is worth having now.
One monitoring caveat specific to ksqlDB’s consumer groups, from our own debugging: lightly loaded persistent queries process records intermittently, so their consumer group statistics look erratic, active, then apparently idle, then active again. On a quiet cluster that reads like something is wrong. Before treating twitchy ksqlDB group stats as a failure, check whether the query is simply keeping pace with sparse input. Ghost-town clusters produce ghost-town metrics.
FAQ
What is ksqlDB used for?
ksqlDB is the streaming SQL layer for Kafka: it expresses stream processing as SQL over topics instead of Java or Scala against the Streams API. CREATE STREAM defines an unbounded sequence of events over a topic, CREATE TABLE defines the current state per key, and a persistent query created from either runs continuously from that point on.
Why does my ksqlDB SELECT never finish?
Because a plain SELECT over a stream is a subscription, not an answer: it keeps emitting rows as events arrive, and there is no end of the data. SQL trained everyone to expect a query to finish, which is exactly the trap. Pull queries against tables return the database-shaped single answer; push queries return the stream. Teach that distinction before the syntax.