Many of the compelling and engaging application experiences we enjoy every day are powered by event-based systems; requesting a ride and watching its progress, communicating with a friend or large group in real time, or connecting our increasingly intelligent devices to our phones and each other. Behind the scenes, similar architectures let developers connect separate services into single systems, or process huge data streams to generate real-time insights. Together, these event-driven architectures and systems are quickly becoming a powerful complement to the relational database and app server models that have been at the core of Internet applications for over twenty years.
At Heroku, we want to make the power of this increasingly important model available to a broader range of developers, allowing them to build evented applications without the cost and complexity of managing infrastructure services. Today we are making a big step towards that goal with the general availability of Apache Kafka on Heroku. Kafka is the industry-leading open source solution for delivering data-intensive applications and managing event streams, and is backed by an active open source community and ecosystem. By combining the innovation of the Kafka community with Heroku’s developer experience and operational expertise, we hope to make this compelling technology even more vibrant and widely adopted, and to enable entirely new use cases and capabilities.
Heroku’s fully managed offering provides all the power, resilience, and scalability of Kafka without the complexity and challenges of operating your own clusters. Here at Heroku, it has been a critical part of our success in delivering fast data processing pipelines for our platform metrics, in creating a robust event stream for our API, and in unifying many of our systems and sources of data. In the ecosystem at large, Kafka has been instrumental to the successful scaling of platforms like LinkedIn, Uber, Pinterest, Netflix, and Square.
Creating your Kafka clusters is as simple as adding the service to a Heroku app:
heroku addons:create heroku-kafka -a sushi-app
Features of Kafka on Heroku include:
Automated Operations: Apache Kafka is a complex, multi-node distributed system, providing power but requiring significant operational care and feeding. With Kafka on Heroku, all of those operational burdens disappear - like other Heroku data services such as Postgres, everything from provisioning, management, and availability is handled automatically. Adding Kafka to your application is now as simple as a single command line.
Simplified Configuration: Optimizing Kafka configuration and resourcing for different applications and use cases is often an art unto itself. Apache Kafka on Heroku offers a set of preconfigured plan options tested and optimized for the most common use cases. Available in both the Common Runtime and within Heroku Private Spaces, Kafka on Heroku can be provisioned with the security and level of network isolation that meets your application needs.
Service Resiliency, Upgrades, and Regions: One of the more powerful capabilities of Apache Kafka on Heroku is the self healing and automated recovery the service provides; if a broker becomes unavailable, the service will automatically replace and re-establish failed elements of the cluster. In addition, Kafka on Heroku clusters can be upgraded in place with no downtime all the way up to multi-terabyte scale. Like other Heroku data services offered in Private Spaces, Kafka on Heroku is available in different regions and geographies, including both east and west coast United States, Dublin, Germany, Japan and Sydney.
Dashboard and Developer Experience: At the core of Apache Kafka on Heroku is a simple provisioning and management experience that fits seamlessly into the rest of the Heroku platform. In addition to a robust set of CLI commands and options, a new Dashboard provides both visibility into the utilization and behavior of a given Kafka cluster, as well as a simple configuration and management UI.
The plans now available are dedicated clusters, optimized for high throughput and high volume. We will continue to extend this range of plans to cover a broader set of needs, and to make evented architectures available to developers of all stripes.
Events are everywhere in modern application development, and they are becoming increasingly important. They’re the lifeblood of everything from activity streams, and IoT devices, to mobile apps, and change data capture. Leveraging these event streams is more important than ever, but doing so requires employing a new set of concepts.
Traditional data architectures focused on transactions and remote procedure calls. These approaches are relatively easy to reason about at certain levels of scale. As systems scale, deal with intense data throughput, and integrate diverse services, developers find that there are very real limits to the transactional and service to service models.
Kafka handles event streams with ease, and provides a solid foundation for designing the logic of your data flows, unifying various data systems, and enabling real-time processing. Kafka also delivers a small but elegant set of abstractions to help developers understand critical facets of event-driven systems like event delivery, ordering, and throughput.
Kafka enables you to easily design and implement architectures for many important use cases, such as:
Kafka makes it easy for systems to accept large volumes of inbound events without putting volatile scaling demands on downstream services.These downstream services can pull from event streams in Kafka when they have capacity, instead of being reactive to the “push” of events. This improves scaling, handling fluctuations in load, and general stability.
Data Pipelines & Analytics
Applications at scale often need analytics or ETL pipelines to get the most value from their data. Kafka’s immutable event streams enable developers to build highly parallel data pipelines for transforming and aggregating data. This means that developers can achieve much faster and more stable data pipelines than would have been possible with batch systems or mutable data.
Many applications move to microservice style architectures as they scale, and run up against the challenges that microservices entail: service discovery, dependencies and ordering of service availability, and service interaction. Applications that use Kafka for communication between services can simplify these design concerns dramatically. Kafka makes it easy for a service to bootstrap into the microservice network, and discover which Kafka topics to send and receive messages on. Ordering and dependency challenges are reduced, and topic based coordination lowers the overhead of service discovery when messages between services are durably managed in Kafka.
Apache Kafka on Heroku is available in all of Heroku’s runtimes, including Private Spaces. Again, to get started, provision Kafka from Heroku Elements or via the command line. We are excited to see what you build with Kafka! If you have any questions or feedback, please let us know at firstname.lastname@example.org or attend our technical session on Nov 3rd to see a demo and get your questions answered!