Today we’re pleased to announce general availability of Heroku Key-Value Store with a number of new features and a more robust developer experience. By giving developers a different data management primitive, we’re helping them meet the needs of building modern, scalable applications. The classic example of using multiple data stores in an application is the e-commerce site that stores its valuable financial information in a relational database while the user session tokens are saved in a key-value store like Redis. This is one of the use cases where Redis has proven to be instrumental in solving problems like caching, queuing and session storage, just to name a few.
In addition to making Heroku Key-Value Store generally available, we’re pleased to introduce something new: Heroku Data Links. The explosion of applications and distributed application architectures -- things like microservices and service-oriented architectures -- means that now more than ever, developers and organizations have valuable data living in disparate data stores. The truly transformative applications and organizations are the ones that can quickly pull all of the data across all of their heterogeneous data stores such as Redis and Postgres, to provide insights back to their own customers or internal users. Heroku Data Links gives developers all of the benefits of data federation across multiple data sources without the hassle of building out a new system. Information is pulled across data stores on-demand via SQL, allowing developers or anyone in the organization to draw insights quickly and easily.
What’s New in Heroku Key-Value Store
Over the course of the public beta, we learned from developers that a great Redis service rests upon three pillars: a great developer experience, security, and high availability.
- Developer Experience: One of our core beliefs is that developers do their best when the tools they use are succinct, elegant and easy to use. We introduced the first of these features at Heroku Key-Value Store public beta, performance analytics and metrics logs. We’re extending the developer experience for GA by introducing command latencies. Latencies in Redis are very critical because data access within Redis should be extremely fast. Command Latencies allows developers to determine bottlenecks in their applications for any number of reasons. For example, a process could be creating key-value pairs that contain large amounts of data, thereby slowing down the entire Redis instance. With the Heroku Key-Value Store dashboard, this is easy to spot.
- Security: Securing your data is something that we, at Heroku, take very seriously. To make sure that developers have a secure Redis to work with, we’re requiring everyone that wants to connect to Redis install the Stunnel buildpack. This will create an SSL connection between the dyno and the Heroku Key-Value Store instance, securing your data in-flight. The benefit of this approach is that it creates a trusted environment for Redis to operate within, requires very little application configuration and no changes to your code. You’re free to use the preferred Redis client library for your programming language knowing that security is already taken care of.
- High Availability: One of the biggest requests we got during the public beta was high availability, and we agree it plays a crucial role in running Redis in production. We have implemented a redundancy strategy that allows for the lowest possible latencies while making HA transparent to the developer. Heroku Key-Value Store uses a master-standby strategy much like Heroku Postgres. In the event of a catastrophic failure of your master Redis instance, Heroku Key-Value Store will automatically fail over to the standby. Your dynos will then get updated Redis connection information ensuring migration of client connections to the new master. We’ll then do the work of setting up another standby and getting it caught up to the new master. We chose this approach because it has been battle tested over the years on our Postgres product, which has successfully performed hundreds of failovers and saved our customers many days of downtime.
We have also been very focused on bringing a full range of pricing options to complement the well rounded developer experience of Heroku Key-Value Store. When you provision a production-tier Heroku Key-Value Store plan, you’ll get everything, including security, performance analytics and high availability. Production-grade plans start at $15 for a 50MB instance and go up to $1,450 for a 10GB. Heroku Key-Value Store can meet the needs of your high scale applications -- if you need something bigger than 10GB, contact us. We’re happy to help.
Even though Heroku Key-Value Store has production-class plans with associated prices, we are going to continue to provide a free tier for anyone looking to prototype or experiment with Redis as part of the application stack. Some of the differences between production plans and our free plan is that the free one does not come with TLS encryption between your app and your Redis instance, high availability guarantees, and some of the dashboard features. The free dev plan is perfect for non-critical hobby apps.
Introducing Heroku Data Links
Data is one of the most valuable assets that an application or business has. Increasingly, this asset can live in multiple Postgres databases or in a different data system entirely. To unlock real value, the data needs to be aggregated, transformed and refined to provide valuable insights to customers and internal users. That process has traditionally encompassed an Extract, Transform, and Load (ETL) program to pull data across all sorts of systems into a data warehouse that aggregated all of the information. The problem with that setup is that you have to invest time and money into a process that isn’t very agile for generating results, and often requires specialized skill. Heroku Data Links uses a single Postgres database to federate data across other data stores, like Redis, to give developers all of the benefits of data federation without the hassle of building out a new system. Paired with Dataclips, this feature can provide sharable data insight within your organization and across data stores.
Under the Hood
To accomplish data aggregation without the costly data extraction processes, we’ve used a native feature of Postgres called Foreign Data Wrappers (FDW). An FDW affords a developer the ability to specify a foreign server, which could be a Redis instance, and the logical representation of foreign tables and map them to tables in the local database. When you write a query against data in a remote database, Postgres will automatically query the remote data in a process transparent to the developer, allowing you to join it with local data. The details of where the data lives and how it’s abstracted in the system are pushed down into the Postgres extension that connects the two data stores. All you need to know is SQL and the table definitions and you can start federating data.
While Postgres has had FDW built-in since version 9.1, we wanted to make FDW easier to use by building a great developer experience on top of it. Heroku Data Links takes care of all of the difficult parts of managing the relationships between data stores. You can add and remove links quickly and easily and we’ll even take care of mapping the tables between the remote data store and the local database. On top of that, we’ll put each link that’s created in its own distinct schema within Postgres so that you don’t have to worry about naming collisions.
Getting Started
Starting today, newly provisioned Heroku Postgres databases, starting at Postgres version 9.4 and above, come with Data Links baked in. We’ve installed the appropriate extensions so that you can use a Heroku Key-Value Store instance as a remote system. Let’s run through an example using a fictional e-commerce application to illustrate how to connect our data stores together.
Linking Heroku Key-Value Store to Heroku Postgres
Let’s say, in our e-commerce application, that we’re storing important financial information in Heroku Postgres and session data in Heroku Key-Value Store. With Heroku Data Links, we can integrate the data from these two systems so that it’s easy to see the profiles of users, living in Postgres, who have signed into the site in the last few hours by joining session data in Redis. In the example below, the local database is HEROKU_POSTGRESQL_CERULEAN while the remote data store is HEROKU_REDIS_PINK:
$ heroku pg:links create HEROKU_REDIS_PINK HEROKU_POSTGRESQL_CERULEAN -a sushi
A Full Data Analysis Stack At Your Disposal
The best part about linking all of your disparate data sources together via Heroku Data Links is that you’ll get access to Dataclips. Heroku Dataclips is a lightweight tool that allows your team to better share, reason about and ask questions of the data you keep in each of your data stores. Fortune 500 companies spend months and years building software and databases to federate data to garner insights. With the one, two punch of Data Links and Dataclips, you’ll have a fully baked analytics solution in a fraction of the time.
Looking to the Future
We’re very excited to bring our operational expertise to running a Redis service that developers can trust as well as level the playing field by reducing the time-to-value for garnering insights with Heroku Data Links. We have many more features and experiences that we plan on building for both products now and into the future. If you have any suggestions of what you’d love to see us build or even just to reach out and tell us how you’re using Data Links, email us!