Go-kit is a distributed programming toolkit for building microservices. It solves the common problems encountered while building distributed systems, so you can focus on your business logic. This article starts with a bit of background on microservices, then guidance on how to get started with Go-kit, including instructions on getting a basic service up and running on Heroku.
A Brief Intro to Microservices
Traditionally, web applications are built using a monolithic approach where the entire application is built, designed, deployed and maintained as a single unit. When working with a monolithic application various problems can arise over time: it’s easy for abstractions to leak across modules, tightly coupling them; different parts of the application may need more processing power than others, forcing you to scale in unpredictable ways; changes involve building and deploying a new version of the entire application; and tests can easily become convoluted and/or take an excruciatingly long time for the full suite to run.
A microservice based design addresses these issues. Applications designed using microservices consist of a set of several small (hence the term “micro”) services cooperating and communicating together. Separation between services is enforced by the service's external API. Each individual micro service can be scaled and deployed separately from the rest.
Design Concerns for Microservices Based Applications
But where monolithic application have one source of logs, one source of metrics, one application to deploy, one API to rate limit, etc, microservice based application have multiple sources. Some of the common concerns of application design that are amplified in a microservices based application are:
Rate limiters enforce upper thresholds on incoming request throughput.
Serialization is the conversion of language specific data structures to a byte stream for presentation to another system. That other system is commonly a browser (json/xml/html) or a database, among others.
Logging is the time-ordered, preferably structured, output from an application and its constituent components.
Metrics are a record of the instrumented parts of your application and includes the aggregated measurements of latency, request counts, health and others.
Circuit breakers prevent thundering herds thereby improving resiliency against intermittent errors.
Request tracing across multiple services is an important tool for diagnosing issues and recreating the state of the system as a whole.
Service discovery allows different services to find each other given known, stable names and the realities of the cloud, where individual systems come and go dynamically and when least expected.
By using a toolkit that addresses these concerns, the implementations across your services becomes standard. This reduces surface area and enforces uniformity in design allowing developers to spend more time on business logic. Go-kit is one such toolkit, comprised of a set of abstractions, encoded into different packages that provide a common set of interfaces for the developer.
Go-kit 101
Let’s make a basic service using Go-kit that keeps a running total of integers passed to it and responds with the current total. To start, we’ll add our business logic, encoded as an interface, with a concrete implementation.
Endpoints in go-kit represent a single RPC and are the fundamental building blocks of clients and servers. Our service implements our Counter interface, while the Endpoint interface is different. (https://en.wikipedia.org/wiki/Adapter_pattern)(https://en.wikipedia.org/wiki/Adapter_pattern) allow a struct that implements one interface to be used where another interface is expected, so an adapter for our service is needed so that it can be used as an endpoint.
Part of exposing a service is handling requests and responses. This is done via a pair of encoder/decoder functions which are then used by go-kit’s http transport. These methods take care of encoding to and decoding from JSON.
Middleware is used to wrap endpoints and add generic, per request functionality to the endpoint. This is done by defining a function that takes an Endpoint and returns a new Endpoint. We use middleware to add logging, metrics and rate limiting. Rate limiting is done using juju’s token bucket and go-kit’s rate limiting middleware. The service’s metric middleware increments a per request counter (request.count), which is exposed as an expvar metric. Lastly, the logging middleware outputs the request path, request id, request and response data, as well as elapsed time. The logging middleware relies on some extractor functions to extract the path and request-id from the *http.Request and add them to the context provided to the middleware and endpoints.
Deploy the complete application by clicking on the “Heroku Button” shown in the README.
Once deployed you can test the application via curl a few times (replace floating-anchorage-23443 with the name of the app you created):
$ curl -X POST -d '{"v":1}' 'https://floating-anchorage-23443.herokuapp.com/add'
{"v":1}
$ curl -X POST -d '{"v":2}' 'https://floating-anchorage-23443.herokuapp.com/add'
{"v":3}
$ curl -X POST -d '{"v":3}' 'https://floating-anchorage-23443.herokuapp.com/add'
{"v":6}
If you execute those curl commands too fast, roughly more than 1 every 2 seconds, the rate limiting middleware will kick in and you’ll get a “rate limit exceeded” error like so:
$ curl -X POST -d '{"v":1}' 'https://floating-anchorage-23443.herokuapp.com/add'
rate limit exceeded
You can also use curl to explore the recorded metrics:
$ curl -s 'https://floating-anchorage-23443.herokuapp.com/debug/vars'
{
…
“request.count”: 4
}
The first curl command above produces a pair of log lines similar to these (viewable with “heroku log”):
$ heroku logs
...
2016-02-17T05:20:43.545994+00:00 heroku[router]: at=info method=POST path="/add" host=floating-anchorage-23443.herokuapp.com request_id=9c248150-03ac-4b24-b5a7-f7c1f0fb05ed fwd="76.115.27.201" dyno=web.1 connect=18ms service=6ms status=200 bytes=143
2016-02-17T05:20:43.546130+00:00 app[web.1]: path=/add request=1 result=1 err=null request_id=9c248150-03ac-4b24-b5a7-f7c1f0fb05ed elapsed=30.206µs
The latter of which is from the service’s logging middleware. The former is from the Heroku Router. A simplified form of request tracing can be achieved by ensuring that all logs across all services that comprise your application include the request id. Heroku encourages customers to drain logs for retention to a logging add-on.
This article covered a very basic app, designed to expose you to Go-kit and its patterns. Build more complex applications, deployed to Heroku Private Spaces, leveraging advanced Go-kit’s features such as: circuit breakers for limiting your exposure to problems with downstream dependencies such as external APIs; load balancers for resolving service endpoints in static and dynamic environments; and Dapper style request tracing using Zipkin.