All posts tagged with kafka


Event-driven architectures are on the rise, in response to fast-moving data and constellations of inter-connected systems. In order to support this trend, last year we released Apache Kafka on Heroku - a gracefully integrated, fully managed, and carefully optimized element of Heroku's platform that is the culmination of years of experience of running many hundreds of Kafka clusters in production and contributing code to the Kafka ecosystem.

Today, we are excited to announce additional plans and pricing in our Kafka offering in order to make Apache Kafka more accessible, and to better support development, testing, and low volume production needs.

At Heroku, we're always working towards improving operational stability with the services we offer. As we recently launched Apache Kafka on Heroku, we've been increasingly focused on hardening Apache Kafka, as well as our automation around it. This particular improvement in stability concerns Kafka's compacted topics, which we haven't talked about before. Compacted topics are a powerful and important feature of Kafka, and as of 0.9, provide the capabilities supporting a number of important features.

Meet the Bug

The bug we had been seeing is that an internal thread that's used by Kafka to implement compacted topics (which we'll explain more of shortly) can die in...

Heroku recently released a managed Apache Kafka offering. As a Node.js developer, I wanted to demystify Kafka by sharing a simple yet practical use case with the many Node.js developers who are curious how this technology might be useful. At Heroku we use Kafka internally for a number of uses including data pipelines. I thought that would be a good place to start.

When it comes to actual examples, Java and Scala get all the love in the Kafka world. Of course, these are powerful languages, but I wanted to explore Kafka from the perspective of Node.js. While there are no technical limitations to using Node.js with Kafka, I was unable to find many examples of their use together in tutorials,...

Many of the compelling and engaging application experiences we enjoy every day are powered by event-based systems; requesting a ride and watching its progress, communicating with a friend or large group in real time, or connecting our increasingly intelligent devices to our phones and each other. Behind the scenes, similar architectures let developers connect separate services into single systems, or process huge data streams to generate real-time insights. Together, these event-driven architectures and systems are quickly becoming a powerful complement to the relational database and app server models that have been at the core of Internet applications for over twenty years.

At Heroku, we...

At Heroku, we're always working towards increased operational stability with the services we offer. As we recently launched the beta of Apache Kafka on Heroku, we've been running a number of clusters on behalf of our beta customers.

Over the course of the beta, we have thoroughly exercised Kafka through a wide range of cases, which is an important part of bringing a fast-moving open-source project to market as a managed service. This breadth of exposure led us to the discovery of a memory leak in Kafka, having a bit of an adventure debugging it, and then contributing a patch to the Apache Kafka community to fix it.

Issue Discovery

For the most part, we’ve seen very few issues...

Browse the blog archives or subscribe to the full-text feed.