Auto-generating a Go API client for Heroku

Editor's note: This is a cross post from Blake Gentry, an engineer at Heroku.

This is a post about the recently announced Heroku Platform API JSON Schema and how I used that schema to write an auto-generated Go client for the API.

Heroku's API team has spent a large part of the past year designing a new version of the platform API. While this is the 3rd incarnation of the API, neither of the two previous versions were publicly documented. In fact, the only documentation on the old APIs that was ever published is the source code of the Heroku Rubygem, which powers the Heroku Toolbelt. That worked fairly well at the time for Heroku's Ruby-centric audience, but it was never ideal, especially since Heroku's developer audience now uses many languages besides Ruby.

Additionally, the first two "versions" of Heroku's API were developed organically as the platform's capabilities evolved, with contributions made by many engineers over the years. The API was built primarily as an interface for the toolbelt, rather than as a product. It was also not properly versioned, as there was no process for managing changes publicly. And because the API was not treated as a product in and of itself, the old APIs are scarred by numerous inconsistencies and lack coherence.

JSON Schema

When the API team embarked on the v3 project, they realized that several of their goals could be made easier if they decided to codify their API design as JSON Schema. Specifically, the JSON schema would make possible:

  • Automatic generation of documentation
  • Automatic server-side validations of input from clients
  • Automatic generation of API client libraries in multiple languages
  • Simpler design of the API by making inconsistencies easier to spot

If you're interested in learning more about JSON Schema, Understanding JSON Schema is an excellent resource.

Getting the schema

The API serves up its own JSON schema programmatically, helping to ensure that the version being provided to users is actually the one that's being used for things like input validation. You can fetch the up-to-date version using curl:

$ curl https://api.heroku.com/schema  -H "Accept: application/vnd.heroku+json; version=3"
{
  "type": [
    "object"
  ],
  "title": "Heroku Platform API",
  "$schema": "http://json-schema.org/draft-04/hyper-schema",
  "properties": {
    "app": {
      "$ref": "/definitions/app"
    },
    ...
    }
  },
  "definitions": {
    "app": {
      "type": [
        "object"
      ],
      "title": "Heroku Platform API - Apps",
      "$schema": "http://json-schema.org/draft-04/hyper-schema",
      "id": "schema/app",
      "description": "An app represents the program that you would like to deploy and run on Heroku.",
      "properties": {
        "web_url": {
          "$ref": "#/definitions/app/definitions/web_url"
        },
        "updated_at": {
          "$ref": "#/definitions/app/definitions/updated_at"
        },
        "stack": {
          "$ref": "#/definitions/stack/definitions/name"
        },
        ...
      },
      "links": [
        {
          "title": "Create",
          "schema": {
            "properties": {
              "stack": {
                "$ref": "#/definitions/stack/definitions/identity"
              },
              "region": {
                "$ref": "#/definitions/region/definitions/identity"
              },
              "name": {
                "$ref": "#/definitions/app/definitions/name"
              }
            }
          },
          "rel": "create",
          "method": "POST",
          "href": "/apps",
          "description": "Create a new app."
        ],
        ...
      }
    },
  ...
  },
  "description": "The platform API empowers developers to automate, extend and combine Heroku with other services."
}

You can view the complete schema by performing the curl request above, or check out a version of it that I saved on 2013-12-26.

More details about the Heroku API team's path to using JSON schema are available in their blog post on the subject.

Generating a client from the schema

A number of Heroku engineers had already taken a crack at generating API clients from the schema:

I spend a great deal of my time working in Go, on hk in particular. hk is a command-line interface for Heroku that interacts heavily with the Heroku API, so a fully-functional Go API client was of great interest to me.

Hand-writing the Golang base client and App model

I've used a number of auto-generated libraries before; they tend to feel like they were made to be used by a computer rather than a human. That's a feeling I wanted to avoid with my client.

I considered a dynamic approach that processed the schema at runtime, but ultimately decided that I wanted to preserve the benefits of Go's static type system. I also wanted to make sure that my code, as much as possible, felt like idiomatic Go, and that it was fully documented in the godoc style.

However, I'd never written a non-trivial piece of auto-generated code before. In order to produce high quality auto-generated code that felt hand-written, I thought it made sense to start by hand writing the base API client with tests. I also hand-coded a single model (the App model) with its own tests.

The base client was derived from client code that was originally baked into hk by Keith Rarick.

I iterated on both the base client and the App model code for a couple of days to make sure that it felt like clean, idiomatic Go code before I attempted to replicate the result with a generator.

Design Notes

One of the things I struggled with while hand-writing the App model was how to cleanly & consistently deal with parameters in API calls. The decision was easy for required parameters: just make them normal, typed function arguments. Simple functions, like getting or deleting a single object, were straightforward to write in this way:

// Info for existing app.
//
// appIdentity is the unique identifier of the App.
func (c *Client) AppInfo(appIdentity string) (*App, error) {
    var app App
    return &app, c.Get(&app, "/apps/"+appIdentity)
}

// Delete an existing app.
//
// appIdentity is the unique identifier of the App.
func (c *Client) AppDelete(appIdentity string) error {
    return c.Delete("/apps/" + appIdentity)
}

Optional parameters, however, required more thought. In some cases, I had to be able to distinguish between the empty value for a parameter and the value not being provided. For example, if the parameter for dyno scale was an int and it was never assigned a value, Go's zero value would be 0. It would be dangerous to assume that the user intended to scale their app to 0. I could have special cased these kinds of situations (maybe using -1 to mean "don't modify"), but that would have made the interface inconsistent.

Instead, I opted to make optional parameters be a pointer of their actual type. In the case of dyno scale, that means that the parameter is scale *int instead of scale int. If scale == nil, then the parameter is omitted altogether from the body of the API call. Fortunately, Go's JSON encoding with omitempty works seamlessly with this approach.

Additionally, while most API actions only had one or two optional arguments, some had three or four (along with required arguments). I don't generally like using functions with so many arguments, especially for optional arguments where I'd have to pass a nil for each one that I didn't want to provide. I also didn't want to use a map[string]interface, as that would mean a loss of type safety.

In the end, I decided to use a dedicated Opts struct for each function with optional arguments, such as AppCreateOpts for the AppCreate function. The caller of the function can then provide nil when they don't want to set any optional parameters, or provide a *AppCreateOpts struct with any desired options set to non-nil values. Here's how that looks:

// Create a new app.
//
// options is the struct of optional parameters for this action.
func (c *Client) AppCreate(options *AppCreateOpts) (*App, error) {
    var appRes App
    return &appRes, c.Post(&appRes, "/apps", options)
}

// AppCreateOpts holds the optional parameters for AppCreate
type AppCreateOpts struct {
    // unique name of app
    Name *string `json:"name,omitempty"`
    // identity of app region
    Region *string `json:"region,omitempty"`
    // identity of app stack
    Stack *string `json:"stack,omitempty"`
}

Finally, I needed to provide a way to utilize the Heroku API's pagination. The API uses Range headers to provide pagination and sorting. Since I already had the notion of an options struct, I decided to use the same pattern for list ranges. The ListRange struct lets users specify any of these options for API calls that list object collections:

// List existing apps.
//
// lr is an optional ListRange that sets the Range options for the paginated
// list of results.
func (c *Client) AppList(lr *ListRange) ([]App, error) {
    req, err := c.NewRequest("GET", "/apps", nil)
    if err != nil {
        return nil, err
    }

    if lr != nil {
        lr.SetHeader(req)
    }

    var appsRes []App
    return appsRes, c.DoReq(req, &appsRes)
}

Generating the Golang API client

Once I had a sample of what I wanted to generate, I set off trying to build a script that could produce that output based on the JSON schema input. Initially I started writing this in Go, but I quickly realized that was a mistake. Go is great for a lot of things, but highly dynamic templates are not its strong point.

Instead, I used an early version of Heroics as a starting point. While the current version of Heroics dynamically ingests the JSON Schema to generate classes and methods at runtime, an early incarnation instead used ERB templates to generate static Ruby code. With this starting point, I began modifying the templates to generate Go code instead of Ruby code.

I started from a position of complete ignorance about the JSON schema specification and tried to figure it out based on what Heroics was already doing. I did not make any attempts to formally parse the JSON schema. Rather, I just iterated on my script until its output converged with my desired code in the App model.

Eventually, I got the results to match 100%, so I began trying to generate the other models as well. Some of them were straightforward, but many had additional edge cases or quirks that required some additional logic in the generator derived from other aspects of the JSON schema. I'd resolve an edge case, then the generator would go a little further before hitting another.

As a side note, I want to call out how much easier this project was because of go fmt. I can generate Go code from a template and not pay any attention to excess whitespace or indentation as a simple go fmt will clean it all up for me.

Auto-generated code can be awesome

In the end, I was able to auto-generate all of the models within about 3-4 days, weighing in at 26 models and 2176 lines (excluding the base client). While the generator is some of the ugliest code I've ever written, given the sheer volume of code involved, writing the generator saved significant time compared to hand-writing the entire client. Using a generator has also made it much easier to update my client when the schema changes, which has happened quite a few times during the Platform API's public beta period. It's also enabled some project-wide refactoring just by changing a couple of lines in the generator. Just re-run the generator, verify the diff, and commit it.

Please check out heroku-go and its documentation. I'd love feedback on the resulting Go code in the form of Github issues and pull requests!

Tags: api, golang, platform

Browse the blog archives or subscribe to the full-text feed.