Blog

A detailed guide to Kafka producer monitoring

A practical guide to Kafka producer metrics, JMX collection, alerting thresholds, and diagnostic scripts for Java-based Kafka producers.

A final goodbye to OperatrIO

2025 is a pivotal moment at Factor House (formally Operatr.IO). We've announced our fundraise and have much more to announce about our roadmap this year. This is why we think that now is the perfect time to do a bit of spring cleaning and retire the io.operatr artifacts for good.

Accelerating incident response: advanced filters, streaming search, and AI-powered queries

Fix streaming data failures faster. Learn how Kpow uses advanced kJQ filtering, BYO AI, and Streaming Search to slash incident response times.

AKHQ: Review, pricing, and best alternatives in 2026

AKHQ review for 2026: features, known limitations, pricing, and the best alternatives for teams that need more than open-source tooling.

Amazon Corretto 11 Memory Issues

A recent move to v2 cgroups by a number of Linux distributions (including Amazon Linux 2022 and Red Hat Enterprise Linux 9) highlights an issue in Amazon Corretto 11 where the JVM process can cause a Docker container to exit with OOMKilled errors.

Apache Kafka 3.2.0: Idempotent Producer Breaking Change

Apache Kafka KIP-679 changes the behaviour of default Producer configuration to enable idempotence by default. This change can cause message production to fail after updating to the 3.2.0 kafka-client libraries.

Apache Kafka 4.3.0: A guide for platform engineers

Kafka 4.3.0 covers broker cordoning, partition size metrics, share group tuning, and tiered storage fixes. Here's what platform engineers need to act on.

Apache Kafka architecture: a complete guide to internals, components, and deployment

A complete guide to Apache Kafka architecture: internals, components, KRaft, replication, consumers, Connect, Streams, and deployment options.

Best Kafka management tools for 2026

Compare the 10 best Kafka management tools for 2026, including Kpow, AKHQ, Conduktor, and Confluent Control Center. Covers pricing, RBAC, and deployment requirements.

Best Kafka monitoring tools for 2026

Compare 12 Kafka monitoring tools for 2026, from enterprise-grade Kpow to open-source AKHQ and Prometheus. Covers deployment, pricing, and key trade-offs.

Best practices for Kafka data observability

12 best practices for Kafka data observability covering consumer lag monitoring, schema enforcement, end-to-end auditing, DLQs, and lineage, with an implementation roadmap.

Beyond JMX: Supercharging Grafana Dashboards with High-Fidelity Metrics

Move beyond raw JMX noise and unlock business-relevant observability for your Kafka environment. This guide explores how to feed high-fidelity, pre-calculated metrics, such as consumer group lag in seconds, directly from Kpow into your Grafana dashboards for proactive capacity planning and incident response.

Beyond Kafka: Sharp Signals from Current London 2025

The real-time ecosystem has outgrown Kafka alone. At Current London 2025, the transition from Kafka Summit was more than a name change — it marked a shift toward streaming-first AI, system-level control, and production-ready Flink. Here's what Factor House saw and learned on the ground.

Beyond Reagent: Migrating to React 19 with HSX and RFX

Introducing two new open sources Clojure UI libraries by Factor House. HSX and RFX are drop-replacements for Reagent and Re-Frame, allowing us to migrate to React 19 while maintaining a familiar developer experience with Hiccup and similar data-driven event model.

Clone to topic for Dead Letter Queues in Apache Kafka

Learn about Clone to Topic, the latest feature available in Kpow 96.2, enabling you to replay Dead Letter Queue (DLQ) records inside a governed UI.

CMAK: Review, pricing, and best alternatives in 2026

CMAK is a free, open-source Kafka admin tool from Yahoo. This review covers features, KRaft limitations, security gaps, and the best alternatives for 2026.

Conduktor: Review, pricing, and best alternatives in 2026

Conduktor review for 2026: pricing, strengths, deployment trade-offs, and how it compares to alternatives for enterprise Kafka governance teams.

Confluent Control Center: Review, pricing, and best alternatives in 2026

An honest technical review of Confluent Control Center in 2026, covering features, deployment, pricing, and the best alternatives for Kafka teams.

Data governance for Kafka: introducing lineage support in Factor Platform

Learn how Factor Platform brings OpenLineage metadata into your Kafka environment, making data ownership, PII classification, and lineage visible by default.

Data Inspect Enhancements in Kpow 94.5

Kpow 94.5 enhances data inspection with comma-separated kJQ Projection expressions, in-browser search, and flexible deserialization options. This release also adds high-performance streaming for large datasets and expands kJQ with new transforms and functions—testable on our new interactive examples page. These updates provide deeper insights and more granular control over your Kafka data streams.

Dead letter queues in Kafka: patterns and pitfalls

How to implement a dead letter queue in Apache Kafka, with Spring Kafka, Connect, and Streams examples, and the production failure modes to avoid.

Defense in depth: unifying RBAC and data policies for transparent governance

Balance Kafka velocity and compliance. Learn how Kpow uses RBAC and Data Policies for safe, self-service production debugging without manual tickets.

Enhanced Under-Replicated Partition Detection in Kpow

Kpow now offers enhanced under-replicated partition (URP) detection for more accurate Kafka health monitoring. Our improved calculation correctly identifies URPs even when brokers are offline, providing a true, real-time view of your cluster's fault tolerance. This helps you proactively mitigate risks and ensure data durability.

Ensuring Your Data Streaming Stack Is Ready for the EU Data Act

The EU Data Act takes effect in September 2025, introducing major implications for teams running Kafka. This article explores what the Act means for data streaming engineers, and how Kpow can help ensure compliance — from user data access to audit logging and secure interoperability.

Factor House expands to Europe

Discover how Factor House's expansion into Germany brings enterprise-grade control and monitoring to European teams running Kafka and Flink.

Factor House Product VPAT

We first published a VPAT in the release notes of Kpow for Apache Kafka v92.4, with a VPAT available to download in every release of Kpow since. Today, we are pleased to announce that we are extending that commitment to all future Factor House product releases - including Flex for Apache Flink and the Factor Platform.

Foundational Kafka data inspection: shaping payloads and optimizing visibility

Stop fighting complex Kafka serialization. Learn how Kpow uses Auto SerDes, kJQ, and transparent queries to streamline data inspection.

From Bootstrap to Blackbird: The Future of Factor House

We are thrilled to announce that Factor House has closed a $5M seed round to accelerate the commercial release of our new product, the Factor Platform. Led by Blackbird Ventures, with OIF Ventures, Flying Fox Ventures, and LaunchVic’s Alice Anderson Fund as partners, this round brings our five-year bootstrapping journey to a happy conclusion and points to a bright future ahead!

How Adidas uses Apache Kafka in production

A deep-dive into Adidas's Kafka architecture — covering observability at 100 billion messages per day, self-service topic provisioning, and custom GoLang tooling.

How Afterpay uses Apache Kafka in production

A deep-dive into Afterpay's Kafka architecture — covering PCI-compliant payment streaming, two-hop AWS PrivateLink isolation, and the Project Teleport cross-region data pipeline.

How Airbnb uses Apache Kafka in production

A deep-dive into Airbnb's Kafka architecture — covering six production systems, 35+ billion daily events, SpinalTap CDC, Flink-based personalisation, and Kafka as a write-ahead log.

How Apple uses Apache Kafka in production

A deep-dive into Apple's Kafka architecture — covering their managed internal platform, Strimzi on EKS, tiered storage, zero-data-movement balancing, and mTLS migration.

How Barclays uses Apache Kafka in production

A deep-dive into Barclays' Kafka architecture — covering dual-environment deployment on AWS and IBM Z-Linux, operating practices, and the broader streaming stack.

How Bytedance uses Apache Kafka in production

How Cash App uses Apache Kafka in production

A deep-dive into Cash App's Kafka architecture — covering the evently-cloud bridge service, per-topic application-layer encryption, and the internal PubSub platform at Block.

How Cloudflare uses Apache Kafka in production

A deep-dive into Cloudflare's Kafka architecture: use cases at trillion-message scale, 14 clusters, internal tooling decisions, and the engineering lessons behind a decade of Kafka operations.

How Datadog uses Apache Kafka in production

A deep-dive into Datadog's Kafka architecture — covering use cases, scale, engineering decisions, and key contributors across hundreds of clusters.

How DoorDash uses Apache Kafka in production

A deep-dive into DoorDash's Kafka architecture — covering the Iguazu event platform, Flink-based ML feature pipelines, self-serve topic governance, and the engineering decisions behind hundreds of billions of daily events.

How Goldman Sachs uses Apache Kafka in production

A deep-dive into Goldman Sachs's Kafka architecture — covering use cases across three divisions, migration to Amazon MSK, resilience design, and key engineering decisions.

How Grab uses Apache Kafka in production

A deep-dive into Grab's Kafka architecture — how the Coban team built a terabyte-per-hour streaming platform serving 300 billion events a week across GrabFood, GrabPay, mobility, and more.

How JPMorgan uses Apache Kafka in production

A deep-dive into JPMorgan Chase's Kafka architecture — covering multi-tenant cluster design, managed Kafka Connect, the Photon Framework, and the engineering decisions behind one of the largest financial services deployments.

How LinkedIn uses Apache Kafka in production

A deep-dive into LinkedIn's Kafka architecture, covering use cases, scale, engineering decisions, and key contributors.

How Netflix uses Apache Kafka in production

A deep-dive into Netflix's Kafka architecture — covering the Keystone pipeline, Data Mesh platform, scale figures from 700 billion to 2 trillion events per day, and the engineering decisions behind it.

How New Relic uses Apache Kafka in production

A deep-dive into New Relic's Kafka architecture — covering use cases, scale, engineering decisions and key contributors.

How Notion uses Apache Kafka in production

A deep-dive into Notion's Kafka architecture — covering use cases, scale, engineering decisions, and key contributors across their data lake and AI pipelines.

How PagerDuty uses Apache Kafka in production

A deep-dive into PagerDuty's Kafka architecture, covering event ingestion, notification scheduling, task execution, and the engineering decisions behind each.

How PayPal uses Apache Kafka in production

A deep-dive into PayPal's Kafka architecture — covering use cases, scale, engineering decisions, and key contributors across a fleet handling 1.3 trillion messages per day.

How Pinterest uses Apache Kafka in production

A deep-dive into Pinterest's Kafka architecture — covering use cases, scale, engineering decisions, and key contributors. From 15 million to 40 million messages per second across 3,000 brokers.

How Reddit uses Apache Kafka in production

A deep-dive into Reddit's Kafka architecture — covering use cases, scale, engineering decisions and key contributors.

How Robinhood uses Apache Kafka in production

A deep-dive into Robinhood's Kafka architecture: use cases, scale, engineering decisions, and key contributors. Learn how Robinhood processes 2.2 million messages per second across equities trading, crypto, fraud detection, and more.

How Salesforce uses Apache Kafka in production

A deep-dive into Salesforce's Kafka architecture — covering use cases, scale, engineering decisions and key contributors across a fleet of 100+ clusters processing 3+ trillion events per day.

How Shopify uses Apache Kafka in production

A deep-dive into Shopify's Kafka architecture — covering CDC at 100,000 records/sec, Kubernetes deployment, the Sarama Go client library, and BFCM scale engineering.

How Spotify used Apache Kafka in production

A deep-dive into Spotify's Kafka architecture — covering their event delivery system, 700K events/second scale, engineering decisions, and why they ultimately migrated to Google Cloud Pub/Sub.

How Tencent uses Apache Kafka in production

A deep-dive into Tencent's Kafka architecture — covering their federated cluster design, 20 trillion messages per day, KIP contributions, and tiered storage at Tencent Cloud.

How The New York Times uses Apache Kafka in production

A deep-dive into The New York Times' Kafka publishing pipeline — covering the Monolog architecture, single-partition design, Kafka Streams usage, and the engineering decisions behind treating Kafka as a permanent content store.

How to monitor Kafka consumer lag: 5 options

Learn what Kafka consumer lag is, why it occurs, and how to monitor it using built-in tools, custom solutions, and Kafka monitoring platforms.

How Uber uses Apache Kafka in production

A deep-dive into Uber's Kafka architecture - covering use cases, scale, engineering decisions, and key contributors. From one region to trillions of messages a day.

How Walmart uses Apache Kafka in production

A deep-dive into Walmart's Kafka architecture — covering real-time inventory, fraud detection, the Customer Data Platform, and the Messaging Proxy Service handling trillions of messages per day.

How Wix uses Apache Kafka in production

A deep-dive into Wix's Kafka architecture: 66 billion daily messages, 2,200+ microservices, the Greyhound SDK, Confluent Cloud migration, and operating 500,000+ partitions across 4 regions.

Improvements to Data Inspect in Kpow 94.3

Kpow's 94.3 release is here, transforming how you work with Kafka. Instantly query topics using plain English with our new AI-powered filtering, automatically decode any message format without manual setup, and leverage powerful new enhancements to our kJQ language. This update makes inspecting Kafka data more intuitive and powerful than ever before.

Introducing Factor House 2.0 🚀

Today we introduce Flex for Apache Flink, and announce the Factor Platform, the future of distributed systems engineering.

Introducing Factor House Docs

We're excited to launch the new Factor House Docs, a unified hub for all our product documentation. Discover key improvements like a completely new task-based structure, interactive kJQ examples, and powerful search, all designed to help you find the information you need, faster than ever. Explore the new home for all things Kpow, Flex, Factor Platform and more.

Introducing Kpow's new API

With our new API, you can now leverage Kpow's capabilities directly from your own tools and platforms, opening up a whole new range of possibilities for integrating Kpow into your existing workflows. Whether you're managing topics, consumer groups, or monitoring Kafka clusters, our API provides a seamless experience that mirrors the functionality of our user interface.

Introduction to Factor House Local

Jumpstart your journey into modern data engineering with Factor House Local. Explore pre-configured Docker environments for Kafka, Flink, Spark, and Iceberg, enhanced with enterprise-grade tools like Kpow and Flex. Our hands-on labs guide you step-by-step, from building your first Kafka client to creating a complete data lakehouse and real-time analytics system. It's the fastest way to learn, prototype, and build sophisticated data platforms.

Join the conversation: Factor House launches open Slack for the real-time data community

Factor House has opened a public Slack for anyone working with streaming data, from seasoned engineers to newcomers exploring real-time systems. This space offers faster peer-to-peer support, open discussion across the ecosystem, and a friendly on-ramp for those just getting started.

Kadeck: Review, pricing, and best alternatives in 2026

Kadeck review for 2026: features, deployment, pricing, and how it compares to AKHQ, Kafbat, Conduktor, and Kpow for Kafka management teams.

Kafbat UI: Review, pricing, and best alternatives in 2026

A practical review of Kafbat, the open-source kafka-ui fork — covering features, deployment, security, pricing, and best alternatives in 2026.

Kafdrop: Review, pricing, and best alternatives in 2026

Kafdrop review for 2026: strengths, limitations, pricing, and the best alternatives for platform and data engineers running production Kafka clusters.

Kafka 4.1 Release: Queues, Stream Groups, and More

Apache Kafka 4.1 has landed: with queue support in preview, improved Kafka Streams coordination, and new security and metrics features, this release marks a major milestone for the future of real-time data systems.

Kafka broker monitoring

How to monitor Kafka brokers: key JMX metrics, alerting thresholds, process monitoring scripts, and common issues with step-by-step diagnosis.

Kafka cluster management: A practical guide for engineers

A practical guide to Kafka cluster management: architecture sizing, day-to-day operations, performance tuning, KRaft migration, and monitoring for production clusters.

Kafka cluster monitoring

What to monitor at the Kafka cluster level: key JMX metrics, multi-broker collection, alerting thresholds, capacity signals, and a health check script.

Kafka consumer monitoring and performance tuning

Learn which Kafka consumer metrics matter most, how to interpret them, and which configuration changes will improve performance and reduce lag.

Kafka dashboard: the features that matter in production

A Kafka dashboard gives you real-time visibility into consumer lag, broker health, and partition state. Here's what to look for and how Kpow delivers it in production.

Kafka Data Management with Kpow: Unlocking Engineering Productivity

Enterprise Kafka adoption promises massive scalability and decoupled agility. However, interacting with complex streaming data at scale often bogs developers down in manual operational friction. By identifying four critical friction points across visibility, velocity, remediation, and compliance, this article introduces a comprehensive data management strategy to eliminate bottlenecks and unlock engineering productivity with Kpow.

Kafka management console: what to look for in a tool

A Kafka management console gives your team full control of topics, consumers, schemas, and connectors from one UI. See what to look for and how Kpow delivers it.

Kafka message key best practices

A technical guide to Kafka message key best practices covering partitioning, ordering guarantees, hot keys, log compaction, and serialization for production systems.

Kafka message size best practice

How large should Kafka messages be in production? Covers sizing tiers, the four-config chain, compression codecs, and patterns for handling payloads above 1 MB.

Kafka monitoring: a complete guide for platform engineers

A practical guide to Kafka monitoring for platform engineers: the metrics that matter, alert thresholds, JVM tuning, consumer lag, and KRaft changes.

Kafka Observability with Kpow: Driving Operational Excellence

Apache Kafka is the central nervous system of the modern enterprise, yet operating it at scale often leads to reactive maintenance cycles. Identifying three critical gaps in context, data quality, and governance, this article introduces a comprehensive strategy to transform reactive troubleshooting into proactive operational excellence with Kpow.

Kafka partition key best practices

How Kafka partition keys work, what makes a good key, and practical guidance on cardinality, hot partitions, compaction, cross-language hashing, and safe key migration.

Kafka scaling best practices: An in-depth primer

A practical guide to scaling Apache Kafka in production, covering partitioning strategy, consumer group design, broker sizing, KRaft migration, and more.

Kafka security architecture: best practices for production

Kafka ships insecure by default. Learn how to build a production-ready Kafka security architecture covering TLS encryption, SASL authentication, ACLs, audit logging, and network isolation.

Kafka topic partition best practices

Size Kafka topic partitions correctly from day one. Covers the throughput formula, the keyed topic asymmetry, KRaft-era limits, and operational best practices.

Kafka UI: The Ultimate Guide

A Kafka UI is a web interface for managing Apache Kafka, giving operators visual control over topics, consumers, brokers, and connectors without the CLI.

KIP-1150 Diskless Topics: Rethinking Storage and Cloud Costs in Kafka

Discover how Kafka's KIP-1150 Diskless Topics aim to bring cloud-native scalability and cost-efficiency by natively utilizing object storage, and what it means for your streaming architecture.

KIP-932 Queues for Kafka: Bridging the Gap Between Streaming and Messaging

Discover how Kafka's KIP-932 Share Groups bring native queue semantics to your event streaming architecture, and the new complexities engineers must manage.

Kpow Community Edition 🚀

Kpow Community Edition is a free, developer focused toolkit for Apache Kafka clusters, schema registries, and connect installations.

Kpow Custom Serdes and Protobuf v4.31.1

This post explains an update in the version of protobuf libraries used by Kpow, and a possible compatibility impact this update may cause to user defined Custom Serdes.

Lenses.io: Review, pricing, and best alternatives in 2026

Lenses.io review for 2026: honest assessment of SQL Studio, deployment complexity, pricing, and when to consider alternatives like Conduktor or Kpow.

Melbourne Kafka x Flink July Meetup Recap: Real-time Data Hosted by Factor House & Confluent

From structuring data streams to spinning up full pipelines locally, our latest Kafka x Flink meetup in Melbourne was packed with hands-on demos and real-time insights. Catch the highlights and what's next.

Operational Transparency: Real-Time Audit Trail Integrated with Webhooks

Operating Kafka without a transparent audit trail creates a critical "Governance Gap", leaving teams blind to administrative changes and vulnerable during incidents. This guide demonstrates how to replace opaque log parsing and restrictive bureaucracy with automated governance by streaming Kpow's real-time audit log via webhooks directly into communication tools like Slack.

Operatr.IO has a new name: Meet Factor House

Meet Factor House, we build Kpow for Apache Kafka

Our Commitment to Engineers

With our funding announcement and the upcoming launch of the Factor Platform, we know some of our existing customers might be wondering: What does this mean for Kpow and Flex? Will we be forced to upgrade? Will prices spike? Keep one thing in mind - at Factor House we're here for engineers.

Rapid Kafka Diagnostics: A Unified Workflow for Root Cause Analysis

The Context Gap caused by fragmented tools hinders effective Kafka monitoring and troubleshooting, as it forces engineers to manually piece together logs and metrics. This guide demonstrates how to close that gap using Kpow's unified workflow to identify the stall, inspect the data, and resolve the incident in a single interface.

RBAC for Kafka: How to Implement and Key Considerations

Learn how to implement Kafka RBAC with practical steps, real-world configuration insights from a hands-on lab, and a clear comparison of RBAC vs ACLs at scale

Redpanda Console: Review, pricing, and best alternatives in 2026

Redpanda Console reviewed for 2026: features, pricing, limitations, and the best alternatives for engineering teams running Apache Kafka or Redpanda.

Releasing Software at Factor House: Our Java Compatibility and Evolution Strategy

At Factor House, delivering reliable software is at the heart of everything we do. A key aspect of this commitment lies in our approach to managing Java compatibility. This blog post outlines our current release process and future plans for evolving Java support, including our approach to deprecating older versions in a way that respects the needs of diverse customer bases.

The Complete Guide to Kafka Change Data Capture (CDC)

Learn how to implement change data capture with Kafka using Debezium. Includes working PostgreSQL CDC examples, architecture patterns, and monitoring.

Top Kafka UI Tools in 2026: A Practical Comparison for Engineering Teams

Honest comparison of Kafka UI tools for enterprise teams. We evaluate AKHQ, Kafbat, Redpanda Console, Conduktor, Confluent Control Center, and Kpow.

Triage, repair, and replay: integrated Kafka remediation workflows

Fix broken Kafka data pipelines fast. Learn how Kpow replaces messy CLI scripts with an intuitive UI to isolate, repair, and re-inject data.

Unified community license for Kpow and Flex

The unified Factor House Community License works with both Kpow Community Edition and Flex Community Edition, meaning one license will unlock both products. This makes it even simpler to explore modern data streaming tools, create proof-of-concepts, and evaluate our products.

Updates to container specifics (DockerHub and Helm Charts)

Discover how our 94.1 release has streamlined DockerHub, Helm Charts, and AWS Marketplace deployments!

Web Accessibility at Factor House

How do skateboards and green suns drive web accessibility at Factor House? Learn why building accessible products is important to us, and how we've changed to ensure that accessibility is embedded in our development process.

What the IBM Confluent acquisition means for Kafka users

IBM's $11B Confluent acquisition raises questions for Kafka users. Assess your lock-in risk across Schema Registry, managed connectors, and operational tooling.