Designing scalable, fault tolerant, and maintainable stream processing systems is not trivial. The Kafka Streams Java library paired with an Apache Kafka cluster simplifies the amount and complexity of the code you have to write for your stream processing system.

Unlike other stream processing systems, Kafka Streams frees you from having to worry about building and maintaining separate infrastructural dependencies alongside your Kafka clusters. However, you still need to worry about provisioning, orchestrating, and monitoring infrastructure for your Kafka Streams applications.

Heroku makes it easy for you to deploy, run, and scale your Kafka Streams applications by using supported...


Jekyll, the static website generator written in Ruby and popularized by GitHub, is a great candidate for being run on Heroku. Originally built to run on GitHub Pages, running Jekyll on Heroku allows you to take advantage of Jekyll’s powerful plugin system to do more than convert Markdown to HTML. On my blog, I have plugins to download my Goodreads current and recently read books and to generate Open Graph images for posts. That said, it’s not straightforward to get up and running on Heroku without using jekyll serve to do the heavy lifting. jekyll serve uses Ruby’s built-in, single threaded web server WEBrick, but a public site should be using a web server more suited for production, like...


Pushing an app to the cloud can feel like launching a probe into space. Once your project is thousands of miles away you can't bang on it with a hammer or replace broken parts when there's a problem. Your debugging efforts must rely on the instrumentation, telemetry, and remote controls included with the app when it was deployed. On Heroku, we've gladly done some of that prep work for you.

Two new Heroku features, Heroku Exec and Language Runtime Metrics, improve your production monitoring, inspecting, and debugging experience on the platform. With Heroku Exec, you can create secure TCP and SSH tunnels into a dyno, which facilitate SSH sessions, port forwarding, remote...


Over the past decade, millions of developers have interacted with the Heroku CLI. In those 10 years, the CLI has gone through many changes. We've changed languages several times; redesigned the plugin architecture; and improved test coverage and the test framework. What follows is the story of our team's journey to build and maintain the Heroku CLI from the early days of Heroku to today.

  1. Ruby (CLI v1-v3)
  2. Go/Node (CLI v4)
  3. Go/Node (CLI v5)
  4. Pure Node (CLI v6)
  5. What's Next?

Ruby (CLI v1-v3)

Our original CLI (v1-v3) was written in Ruby and served us well for many years. Ruby is a great, expressive language for building CLIs, however, we started experiencing enough problems that...


The Heroku Connect team ran into problems with existing task scheduling libraries. Because of that, we wrote RedBeat, a Celery Beat scheduler that stores scheduled tasks and runtime metadata in Redis. We’ve also open sourced it so others can use it. Here is the story of why and how we created RedBeat.

Background

Heroku Connect, makes heavy use of Celery to synchronize data between Salesforce and Heroku Postgres. Over time, our usage has grown, and we came to rely more and more heavily on the Beat scheduler to trigger frequent periodic tasks. For a while, everything was running smoothly, but as we grew cracks started to appear. Beat, the default Celery scheduler, began to behave...


Browse the archives for engineering or all blogs Subscribe to the RSS feed for engineering or all blogs.