Posts by Mark McGranaghan

The Heroku HTTP API Toolchain

Today we’re open sourcing the toolchain Heroku uses to design, document, and consume our HTTP APIs. We hope this shows how Heroku thinks about APIs and gives you new tools to create your own.

This toolchain includes:

  • An HTTP API design guide, describing how we structure both internal and public-facing APIs and document them using the JSON Schema standard.

  • A tool for working with JSON schemas and using them to generate API documentation.

  • Ruby and Go client code generators for APIs with JSON schemas.

Here’s some more information about these things, how we use them at Heroku, and an explanation of how you can try them yourself.

JSON Schema Foundation

We’ve developed the toolchain around...

Continue reading »

Incident Response at Heroku

As a service provider, when things go wrong you try to get them fixed as quickly as possible. In addition to technical troubleshooting, there’s a lot of coordination and communication that needs to happen in resolving issues with systems like Heroku’s.

At Heroku we’ve codified our practices around these aspects into an incident response framework. Whether you’re just interested in how incident response works at Heroku, or looking to adopt and apply some of these practices for yourself, we hope you find this inside look helpful.

Incident Response and the Incident Commander Role

We describe Heroku’s incident response framework below. It’s based on the Incident Command System used in...

Continue reading »

Rails Security Vulnerability

A serious security vulnerability has been found in the Ruby on Rails framework. This exploit affects nearly all applications running Rails and a patch has been made available.

Rails developers can get a full list of all your affected Heroku applications by following instructions here. Please address this security vulnerability by immediately upgrading your affected apps to any of the safe versions of Rails listed below. The following Rails versions have been patched and deemed safe from this exploit:

  • 3.2.11
  • 3.1.10
  • 3.0.19
  • 2.3.15

If you do not upgrade, an attacker can trivially gain access to your application, its data, and run arbitrary code or commands. Heroku recommends upgrading to a...

Continue reading »

Tuesday Postmortem

Tuesday was not a good day for Heroku and as a result it was not a good day for our customers. I want to take the time to explain what happened, how we addressed the problem, and what we’re doing in the future to keep it from happening again.

Over the past few weeks we have seen unprecedented growth in the rate of new applications being added to the platform. This growth has exacerbated a problem with our internal messaging systems that we’ve known about and been working to address. Unfortunately, the projects that we have underway to address the problem were planned based on previous growth rates and are not yet complete.

A slowdown in our internal messaging systems caused a...

Continue reading »

Browse the blog archives, subscribe to the full-text feed, or visit the engineering blog.