Securing Celery on Heroku

Celery is by far the most popular library in Python for distributing asynchronous work using a task queue. If you're building a Python web app, chances are you already use it to send email, perform API integrations, etc. Many people choose Redis as their message broker of choice because it's dead simple to set up: provision a Redis addon, use its envrionment variable as your BROKER_URL, and you're done. But the simplicity of Redis comes at a cost. Redis does not currently support SSL, and it doesn't seem like that's going to change any time soon. Because Heroku add-ons communicate over the public web, that means the contents of Celery jobs are traveling unencrypted between dynos and Redis.

Luckily, this is a solvable problem. At Heroku, we often use Fernet to symmetrically encrypt information over the wire. Kombu, Celery's messaging library, supports custom serializers, giving us control over how Celery jobs are turned into byte strings to be sent to the message broker and vice versa. We've wrapped all of the default serializers that come with Kombu in Fernet encryption (provided by Cryptography) and published them in an open source library called kombu-fernet-serializers.

To use the library, add kombu-fernet-serializers to requirements.txt, store your Fernet key in an environment variable called KOMBU_FERNET_KEY, and tell Celery to use one of the serializers it supplies via Setuptools entry-points:

import os
from celery import Celery
from kombu_fernet.serializers.json import MIMETYPE

app = Celery('tasks', broker=os.environ['REDIS_URL'])
app.conf.update(
    CELERY_TASK_SERIALIZER='fernet_json',
    CELERY_ACCEPT_CONTENT=[MIMETYPE],
)

Now you can take advantage of the ease of configuration Redis boasts without having to worry about leaking sensitive information.

More from the author

Browse the archives for engineering or all blogs Subscribe to the RSS feed for engineering or all blogs.