Today we are happy to announce early access to Heroku Kafka. We think Kafka is interesting and exciting because it provides a powerful and scalable set of primitives for reasoning about, building, and scaling systems that can handle high volumes and velocities of data. Heroku Kafka makes Kafka more accessible, reliable, and easy to integrate into your applications.
What is Kafka?
Apache Kafka is a distributed commit log for fast, fault-tolerant communication between producers and consumers using message based topics. Kafka provides the messaging backbone for building a new generation of distributed applications capable of handling billions of events and millions of transactions.
At Heroku we use Kafka to create more robust stream processing systems for our operational metrics, and to bring change data capture semantics to the event bus behind our API. It has been an amazingly powerful tool for the customers we’ve worked with over the course of our private beta, enabling them to synchronize data across systems, manage high-volume inbound event streams, build simplified service oriented architectures, and more.
Kafka the Heroku Way
Heroku has years of experience creating data services as products. We have already productized great open source projects like Postgres and Redis, offering them to all developers as Heroku services with powerful features and a unique developer experience. With Heroku Kafka, we extend this capability to a new type of particularly complex data service: distributed systems.
Extending the Heroku developer experience to Kafka also means creating carefully crafted CLI and Web tools that make it easy to configure and manage Kafka alongside your Heroku applications. These tools provide you the metrics, logging and insights you need to tune your integration, making it easier to focus on the logic of your products, and to scale with ease. Kafka producers and consumers can also be developed, deployed, monitored, and scaled as Heroku applications, providing a fully integrated development experience.
This initial release represents the first step, and we look forward to adding new functionality as we rollout Heroku Kafka fully in the coming months.
Get Early Access
Heroku Kafka is in early access; you can get on the list to participate in the beta today and we’ll be inviting developers on a rolling basis. We can’t wait to see what you build with a new ecosystem of Heroku data services to power your applications.