Posts by Scott Persinger

Powering the Heroku Platform API: A Distributed Systems Approach Using Streams and Apache Kafka

We recently launched Apache Kafka on Heroku into beta. Just like we do with Heroku Postgres, our internal engineering teams have been using our Kafka service to power a number of our internal systems.

The Big Idea

The Heroku platform comprises a large number of independent services. Traditionally we’ve used HTTP calls to communicate between these services. While this approach is simple to implement and easy to reason about, it has a number of drawbacks. Synchronous calls mean that the top-level request time will be gated by the slowest backend component. Also, internal API calls create tight point-to-point couplings between services that can become very brittle over time.


Continue reading »

Browse the blog archives, subscribe to the full-text feed, or visit the engineering blog.