[{"title":"How to Use pgvector for Similarity Search on Heroku Postgres","content":"\u003ch2 class='anchored'\u003e\n  \u003ca name='introducing-pgvector-for-heroku-postgres' href='#introducing-pgvector-for-heroku-postgres'\u003eIntroducing pgvector for Heroku Postgres\u003c/a\u003e\n\u003c/h2\u003e\n\n\u003cp\u003eOver the past few weeks, we worked on adding \u003ca href=\"https://github.com/pgvector/pgvector\"\u003epgvector\u003c/a\u003e as an extension on Heroku Postgres. We\u0026#39;re excited to release this feature, and based on the feedback on \u003ca href=\"https://github.com/heroku/roadmap/issues/156\"\u003eour public roadmap\u003c/a\u003e, many of you are too. We want to share a bit more about how you can use it and how it may be helpful to you. \u003c/p\u003e\n\n\u003cp\u003eAll \u003ca href=\"https://devcenter.heroku.com/articles/heroku-postgres-plans#plan-tiers\"\u003eStandard-tier or higher\u003c/a\u003e databases running Postgres 15 now support the \u003ca href=\"https://devcenter.heroku.com/articles/heroku-postgres-extensions-postgis-full-text-search#pgvector\"\u003e\u003ccode\u003epgvector\u003c/code\u003e extension\u003c/a\u003e. You can get started by running \u003ccode\u003eCREATE EXTENSION vector;\u003c/code\u003e in a client session. Postgres 15 has been the default version on Heroku Postgres since March 2023.  If you\u0026#39;re on an older version and want to use pgvector, \u003ca href=\"https://devcenter.heroku.com/articles/upgrading-heroku-postgres-databases\"\u003eupgrade\u003c/a\u003e to Postgres 15.\u003c/p\u003e\n\n\u003cp\u003eThe extension adds the vector data type to Heroku Postgres along with additional functions to work with it. Vectors are important for working with large language models and other machine learning applications, as the \u003ca href=\"https://huggingface.co/blog/getting-started-with-embeddings#understanding-embeddings\"\u003eembeddings\u003c/a\u003e generated by these models are often output in vector format. Working with vectors lets you implement things like similarity search across these embeddings. See our \u003ca href=\"https://blog.heroku.com/pgvector-launch#understanding-pgvector-and-its-significance\"\u003elaunch blog\u003c/a\u003e for more background into what pgvector is, its significance, and ideas for how to use this new data type.\u003c/p\u003e\n\u003ch2 class='anchored'\u003e\n  \u003ca name='an-example-word-vector-similarity-search' href='#an-example-word-vector-similarity-search'\u003eAn Example: Word Vector Similarity Search\u003c/a\u003e\n\u003c/h2\u003e\n\n\u003cp\u003eTo show a simple example of how to generate and save vector data to your Heroku database, I\u0026#39;m using the \u003ca href=\"https://wikipedia2vec.github.io/wikipedia2vec/\"\u003eWikipedia2Vec\u003c/a\u003e pretrained embeddings. However, you can train your own embeddings or use other models providing embeddings via API, like \u003ca href=\"https://huggingface.co/blog/getting-started-with-embeddings\"\u003eHuggingFace\u003c/a\u003e or \u003ca href=\"https://openai.com/\"\u003eOpenAI\u003c/a\u003e. The model you want to use depends on the type of data you\u0026#39;re working with. There are models for tasks like computing sentence similarities, searching large texts, or performing image classification. Wikipedia2Vec uses a \u003ca href=\"https://en.wikipedia.org/wiki/Word2vec\"\u003eWord2vec\u003c/a\u003e algorithm to generate vectors for individual words, which maps similar words close to each other in a continuous vector space. \u003c/p\u003e\n\n\u003cp\u003eI like animals, so I want to use Wikipedia2Vec to group similar animals. I’m using the vector embeddings of each animal and the distance between them to find animals that are alike.\u003c/p\u003e\n\n\u003cp\u003eIf I want to get the embedding for a word from Wikipedia2Vec, I need to use a model. I downloaded one from the \u003ca href=\"https://wikipedia2vec.github.io/wikipedia2vec/pretrained/\"\u003epretrained embeddings\u003c/a\u003e on their website. Then I can use their Python module and the function \u003ccode\u003eget_word_vector\u003c/code\u003e as follows:\u003c/p\u003e\n\n\u003cpre\u003e\u003ccode\u003efrom wikipedia2vec import Wikipedia2Vec\nwiki2vec = Wikipedia2Vec.load(\u0026#39;enwiki_20180420_100d.pkl\u0026#39;)\nwiki2vec.get_word_vector(\u0026#39;llama\u0026#39;)\n\u003c/code\u003e\u003c/pre\u003e\n\n\u003cp\u003eThe output of the vector looks like this:\u003c/p\u003e\n\n\u003cpre\u003e\u003ccode\u003ememmap([-0.15647224,  0.04055957,  0.48439676, -0.22689971, -0.04544162,\n        -0.06538601,  0.22609918, -0.26075622, -0.7195759 , -0.24022003,\n         0.1050799 , -0.5550985 ,  0.4054564 ,  0.14180332,  0.19856507,\n         0.09962048,  0.38372937, -1.1912689 , -0.93939453, -0.28067762,\n         0.04410955,  0.43394643, -0.3429818 ,  0.22209083, -0.46317756,\n        -0.18109794,  0.2775289 , -0.21939017, -0.27015808,  0.72002393,\n        -0.01586861, -0.23480305,  0.365697  ,  0.61743397, -0.07460125,\n        -0.10441436, -0.6537417 ,  0.01339269,  0.06189647, -0.17747395,\n         0.2669941 , -0.03428648, -0.8533792 , -0.09588563, -0.7616592 ,\n        -0.11528812, -0.07127796,  0.28456485, -0.12986512, -0.8063386 ,\n        -0.04875885, -0.27353695, -0.32921   , -0.03807172,  0.10544889,\n         0.49989182, -0.03783042, -0.37752548, -0.19257008,  0.06255971,\n         0.25994852, -0.81092316, -0.15077794,  0.00658835,  0.02033841,\n        -0.32411653, -0.03033727, -0.64633304, -0.43443972, -0.30764043,\n        -0.11036412,  0.04134453, -0.26934972, -0.0289086 , -0.50319433,\n        -0.0204528 , -0.00278326,  0.36589545,  0.5446438 , -0.10852882,\n         0.09699931, -0.01168614,  0.08618425, -0.28925297, -0.25445923,\n         0.63120073,  0.52186656,  0.3439454 ,  0.6686451 ,  0.1076297 ,\n        -0.34688494,  0.05976971, -0.3720558 ,  0.20328045, -0.485623  ,\n        -0.2222396 , -0.22480975,  0.4386788 , -0.7506131 ,  0.14270408],\n       dtype=float32)\n\u003c/code\u003e\u003c/pre\u003e\n\n\u003cp\u003eTo get your vector data into your database:\u003c/p\u003e\n\n\u003col\u003e\n\u003cli\u003eGenerate the embeddings.\u003c/li\u003e\n\u003cli\u003eAdd a column to your database to store your embeddings.\u003c/li\u003e\n\u003cli\u003eSave the embeddings to the database.\u003c/li\u003e\n\u003c/ol\u003e\n\n\u003cp\u003eI already have the embeddings from Wikipedia2Vec, so let’s walk through preparing my database and saving them. When creating a vector column, it\u0026#39;s necessary to declare a length for it, so check and see the length of the embedding the model outputs. In my case, the embeddings are 100 numbers long, so I add that column to my table.\u003c/p\u003e\n\n\u003cpre\u003e\u003ccode\u003eCREATE TABLE animals(id serial PRIMARY KEY, name VARCHAR(100), embedding VECTOR(100));\n\u003c/code\u003e\u003c/pre\u003e\n\n\u003cp\u003eFrom there, save the items you\u0026#39;re interested in to your database. You can do it directly in SQL:\u003c/p\u003e\n\n\u003cpre\u003e\u003ccode\u003eINSERT INTO animals(name, embedding) VALUES (\u0026#39;llama\u0026#39;, \u0026#39;[-0.15647223591804504, \n…\n-0.7506130933761597, 0.1427040845155716]\u0026#39;);\n\u003c/code\u003e\u003c/pre\u003e\n\n\u003cp\u003eBut you can also use your \u003ca href=\"https://devcenter.heroku.com/articles/connecting-heroku-postgres\"\u003efavorite programming language\u003c/a\u003e along with a Postgres client and a \u003ca href=\"https://github.com/pgvector/pgvector#languages\"\u003epgvector library\u003c/a\u003e. For this example, I used Python, \u003ca href=\"https://github.com/psycopg/psycopg\"\u003epsycopg\u003c/a\u003e, and \u003ca href=\"https://github.com/pgvector/pgvector-python\"\u003epgvector-python\u003c/a\u003e. Here I\u0026#39;m using the pretrained embedding file to generate embeddings for a list of animals I made, \u003ccode\u003evaleries-animals.txt\u003c/code\u003e,  and save them to my database.\u003c/p\u003e\n\n\u003cpre\u003e\u003ccode\u003eimport psycopg\nfrom pathlib import Path\nfrom pgvector.psycopg import register_vector\nfrom wikipedia2vec import Wikipedia2Vec\n\nwiki2vec = Wikipedia2Vec.load(\u0026#39;enwiki_20180420_100d.pkl\u0026#39;)\nanimals = Path(\u0026#39;valeries-animals.txt\u0026#39;).read_text().split(\u0026#39;\\n\u0026#39;)\n\nwith psycopg.connect(DATABASE_URL, sslmode=\u0026#39;require\u0026#39;, autocommit=True) as conn:\n    register_vector(conn)\n    cur = conn.cursor()\n    for animal in animals:\n        cur.execute(\u0026quot;INSERT INTO animals(name, embedding) VALUES (%s, %s)\u0026quot;, (animal, wiki2vec.get_word_vector(animal)))\n\u003c/code\u003e\u003c/pre\u003e\n\n\u003cp\u003eNow that I have the embeddings in my database, I can use pgvector\u0026#39;s functions to query them. The extension includes functions to calculate Euclidean distance (\u003ccode\u003e\u0026lt;-\u0026gt;\u003c/code\u003e), cosine distance (\u003ccode\u003e\u0026lt;=\u0026gt;\u003c/code\u003e), and inner product (\u003ccode\u003e\u0026lt;#\u0026gt;\u003c/code\u003e). You can use all three for \u003ca href=\"https://developers.google.com/machine-learning/clustering/similarity/measuring-similarity\"\u003ecalculating similarity\u003c/a\u003e between vectors. Which one you use depends on \u003ca href=\"https://cmry.github.io/notes/euclidean-v-cosine\"\u003eyour data as well as your use case\u003c/a\u003e.\u003c/p\u003e\n\n\u003cp\u003eHere I\u0026#39;m using Euclidean distance to find the five animals closest to a shark:\u003c/p\u003e\n\n\u003cpre\u003e\u003ccode\u003e=\u0026gt; SELECT name FROM animals WHERE name != \u0026#39;shark\u0026#39; ORDER BY embedding \u0026lt;-\u0026gt; (SELECT embedding FROM animals WHERE name = \u0026#39;shark\u0026#39;) LIMIT 5;\n name \n-----------\n crocodile\n dolphin\n whale\n turtle\n alligator\n(5 rows)\n\u003c/code\u003e\u003c/pre\u003e\n\n\u003cp\u003eIt works! It\u0026#39;s worth noting that the model that we used is based on words appearing together in Wikipedia articles, and different models or source corpuses likely yield different results. The results here are also limited to the hundred or so animals that I added to my database.\u003c/p\u003e\n\u003ch2 class='anchored'\u003e\n  \u003ca name='pgvector-optimization-and-performance-considerations' href='#pgvector-optimization-and-performance-considerations'\u003epgvector Optimization and Performance Considerations\u003c/a\u003e\n\u003c/h2\u003e\n\n\u003cp\u003eAs you add more vector data to your database, you may notice performance issues or slowness in performing queries. You can index vector data like other columns in Postgres, and pgvector provides a few ways to do so, but there are some important considerations to keep in mind:\u003c/p\u003e\n\n\u003cul\u003e\n\u003cli\u003eAdding an index causes pgvector to switch to using approximate nearest neighbor search instead of exact nearest neighbor search, possibly causing a difference in query results.\u003c/li\u003e\n\u003cli\u003eIndexing functions are based on distance calculations, so create one based on the calculation you plan to rely on the most in your application.\u003c/li\u003e\n\u003cli\u003eThere are two index types supported, IVFFlat and HNSW. Before you add an IVFFlat index, make sure you have some data in your table for better recall.\u003c/li\u003e\n\u003c/ul\u003e\n\n\u003cp\u003eCheck out the \u003ca href=\"https://github.com/pgvector/pgvector#indexing\"\u003epgvector documentation\u003c/a\u003e for more information on indexing and other performance considerations.\u003c/p\u003e\n\u003ch2 class='anchored'\u003e\n  \u003ca name='collaborate-and-share-your-pgvector-projects' href='#collaborate-and-share-your-pgvector-projects'\u003eCollaborate and Share Your pgvector Projects\u003c/a\u003e\n\u003c/h2\u003e\n\n\u003cp\u003eNow that pgvector for Heroku Postgres is out in the world, we\u0026#39;re really excited to hear what you do with it! One of pgvector\u0026#39;s great advantages is that it lets vector data live alongside all the other data you might already have in Postgres. You can add an embedding column to your existing tables and start experimenting. Our \u003ca href=\"https://blog.heroku.com/pgvector-launch\"\u003elaunch blog\u003c/a\u003e for this feature includes a lot of ideas and possible use cases for how to use this new tool, and I\u0026#39;m sure you can come up with many more. If you have questions, our \u003ca href=\"https://help.heroku.com/\"\u003eSupport team\u003c/a\u003e is available to assist. Don\u0026#39;t forget you can share your solutions using the \u003ca href=\"https://devcenter.heroku.com/articles/heroku-button\"\u003eHeroku Button\u003c/a\u003e on your repo. If you feel like blogging on your success, tag us on social media and we would love to read about it!\u003c/p\u003e\n","published_at":"2023-11-15T18:42:51.442Z","permalink":"https://blog.heroku.com/pgvector-for-similarity-search-on-heroku-postgres","tags":[],"summary":"Introducing pgvector for Heroku Postgres. The extension adds the vector data type to Heroku Postgres along with additional functions to work with it."},{"title":"Router 2.0: The Road to Beta","content":"\u003cp\u003eLast month, Heroku announced the \u003ca href=\"https://devcenter.heroku.com/changelog-items/2682\"\u003ebeta release of Router 2.0\u003c/a\u003e, the new Common Runtime router! \u003c/p\u003e\n\n\u003cp\u003eAs part of our commitment to infrastructure modernization, Heroku is making upgrades to the \u003ca href=\"https://devcenter.heroku.com/articles/dyno-runtime#common-runtime\"\u003eCommon Runtime\u003c/a\u003e routing layer. The \u003ca href=\"https://devcenter.heroku.com/changelog-items/2682\"\u003ebeta release of Router 2.0\u003c/a\u003e is an important step along this journey. We’re excited to give you an inside look at all we’ve been doing to get here. \u003c/p\u003e\n\n\u003cp\u003eIn both the Common Runtime and \u003ca href=\"https://devcenter.heroku.com/articles/dyno-runtime#private-spaces-runtime\"\u003ePrivate Spaces\u003c/a\u003e, the \u003ca href=\"https://devcenter.heroku.com/articles/how-heroku-works#http-routing\"\u003eHeroku router\u003c/a\u003e is responsible for serving requests to customers’ web dynos. In 2024, Router 2.0 will replace the existing Common Runtime router. We’re being transparent about this project so that you, our customers, are motivated to try out Router 2.0 now, while it’s in beta. As an early adopter, you can help us validate that things are working as they should, particularly for \u003cem\u003eyour\u003c/em\u003e apps and \u003cem\u003eyour\u003c/em\u003e use cases. You’ll also be first in line to try out the new features we’re planning to add, like \u003ca href=\"https://github.com/heroku/roadmap/issues/34\"\u003eHTTP/2\u003c/a\u003e.\u003c/p\u003e\n\u003ch2 class='anchored'\u003e\n  \u003ca name='why-a-new-router' href='#why-a-new-router'\u003eWhy a New Router?\u003c/a\u003e\n\u003c/h2\u003e\n\n\u003cp\u003eNow, you may be asking, why build a new router instead of improving the existing one? Our primary motivator has been faster and safer delivery of new routing features for our customers. For a couple of reasons, this has been difficult to achieve with the Common Runtime’s legacy routing layer.\u003c/p\u003e\n\n\u003cp\u003eThe current Common Runtime router is written in Erlang. It’s built around a \u003ca href=\"https://blog.heroku.com/vegur-free-software\"\u003ecustom HTTP server library\u003c/a\u003e that supports Heroku-specific features, such as \u003ca href=\"https://devcenter.heroku.com/articles/error-codes\"\u003eH-codes\u003c/a\u003e, \u003ca href=\"https://devcenter.heroku.com/articles/eco-dyno-hours#dyno-sleeping\"\u003edyno sleeping\u003c/a\u003e, and \u003ca href=\"https://devcenter.heroku.com/articles/http-routing#heroku-router-log-format\"\u003erouter logs\u003c/a\u003e. For over 10 years, this router, dubbed “Hermes” internally, has served all requests to Heroku’s Common Runtime. At the time of Hermes’ launch, Erlang was an appropriate choice since the language places emphasis on concurrency, scalability, and fault tolerance. In addition, Erlang offers a powerful process introspection toolchain that has served our networking engineers well when \u003ca href=\"https://blog.heroku.com/erlang-in-anger\"\u003edebugging in-memory state issues\u003c/a\u003e. Our engineers embraced the language fully, also choosing to write the previous version of our logging system, \u003ca href=\"https://blog.heroku.com/logging-on-heroku\"\u003eLogplex\u003c/a\u003e, in Erlang. \u003c/p\u003e\n\n\u003cp\u003eHowever, as the years passed, development on the Hermes codebase proved difficult. The popularity of Erlang within Heroku began to taper off. The open-source and internal libraries that Hermes depends on stopped receiving the volume of contributions they once had. Productivity declined due to these factors, making significant router upgrades risky. After a few failed upgrade attempts, our team decided to pin the software versions of relevant Erlang components. This action wasn’t without trade-offs. Being pinned to an old version of Erlang became a blocker to delivering now common-place features like \u003ca href=\"https://github.com/heroku/roadmap/issues/34\"\u003eHTTP/2\u003c/a\u003e. Thus, we decided to put Hermes into maintenance mode and focus on its replacement.\u003c/p\u003e\n\u003ch2 class='anchored'\u003e\n  \u003ca name='choosing-a-language' href='#choosing-a-language'\u003eChoosing a Language\u003c/a\u003e\n\u003c/h2\u003e\n\n\u003cp\u003eBefore kicking off design sessions, our team discussed what broader goals we had for the replacement. In establishing our priorities, the team came to a consensus around three main goals:\u003c/p\u003e\n\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eWrite the router in a language everyone on our team knows well\u003c/strong\u003e. With Erlang knowledge limited to just a couple of engineers on the team, we wanted to rewrite the router in a different language. That language had to be something our team already knew well.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eWrite the router in a language with a strong open-source community.\u003c/strong\u003e A robust community unlocks the ability to quickly adopt new specs, write features, fix bugs, and respond to CVEs. It also expands the candidate pool when it comes time to hire new engineers.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eShare as much code as possible between the Common Runtime and Private Spaces routers.\u003c/strong\u003e Since the Common Runtime and Private Spaces routers share most of the same features, there’s no reason for the codebases to differ much. Additionally, it’s faster and easier to deliver a feature if we only have to write it once.\u003c/li\u003e\n\u003c/ul\u003e\n\n\u003cp\u003eWith these goals in mind, the language to choose for Router 2.0 was clear — Go.\u003c/p\u003e\n\n\u003cp\u003eNot only is the Private Spaces router already written in Go, but the language has become our standard choice for developing new components of Heroku’s runtime. This story isn’t at all unique. Many others in the DevOps and cloud hosting world today have chosen Go for its performance, built-in concurrency handling, automatic garbage collection — the list goes on. Simply put, it’s a language designed specifically for building big dynamic distributed systems. Because of these factors, the Go community outside and within Heroku has flourished, with Go expertise in abundance across our runtime engineering teams.\u003c/p\u003e\n\n\u003cp\u003eToday, by writing Router 2.0 in Go, we’re creating a piece of software to which everyone on our team can contribute. Furthermore, by doubling down on the language of the Private Spaces router, we unify the source code and routing behavior of these two products. Historically, these codebases have been entirely distinct, meaning that any implementation our engineers introduce must be written twice. To combat this, we’ve extracted the common functionality of the two routers into an internal HTTP library. With a unified codebase, the delivery of features and fixes becomes faster and simpler, reducing the cognitive burden on our engineers who operate and maintain the routers.\u003c/p\u003e\n\n\u003cp\u003eDeveloping the router is only half the story, though. The other half is about introducing this service to the world as safely and seamlessly as possible.\u003c/p\u003e\n\u003ch2 class='anchored'\u003e\n  \u003ca name='architecture' href='#architecture'\u003eArchitecture\u003c/a\u003e\n\u003c/h2\u003e\n\n\u003cp\u003eYou may recall that back in 2021, Heroku announced the completion of an \u003ca href=\"https://blog.heroku.com/faster-dynos-for-all\"\u003einfrastructure upgrade\u003c/a\u003e to the Common Runtime that brought customers better performing dynos and lower request latencies. This upgrade involved an extensive migration from our old, “classic” cloud environment to our more performant and secure “sharded” environment. We wanted to complete this migration without disrupting any active traffic or asking customers to change their DNS setups. To do this, our engineers put an \u003ca href=\"https://www.geeksforgeeks.org/open-systems-interconnection-model-osi/\"\u003eL4\u003c/a\u003e reverse proxy in front of Hermes, straddling the classic and sharded environments. The idea was to slowly shift traffic over to the sharded environments, with the L4 proxy splitting connections to both the classic and the new “in-shard” Hermes instances.\u003c/p\u003e\n\n\u003cp\u003eAlso a part of this migration, TLS termination on custom domains was transitioned from Hermes to the L4 proxy.\u003c/p\u003e\n\n\u003cp\u003e\u003cimg src=\"https://heroku-blog-files.s3.amazonaws.com/posts/1698271166-IMG_2180.jpeg\" alt=\"IMG_2180\"\u003e\nThis L4 proxy is the component that has formed the basis for Router 2.0. Over the past year, our networking team has been developing an L7 router to sit in-memory behind the L4 proxy. Today, the L4 proxy + Router 2.0 process runs alongside Hermes, communicating over the \u003ccode\u003elocalhost\u003c/code\u003e network on our router instances. Putting these two processes side by side, instead of on separate hosts, means we limit the number of network hops between clients and backend dynos.\u003c/p\u003e\n\u003ch3 class='anchored'\u003e\n  \u003ca name='the-strangler-pattern' href='#the-strangler-pattern'\u003eThe Strangler Pattern\u003c/a\u003e\n\u003c/h3\u003e\n\n\u003cp\u003eFor apps still on the default routing path, client connections are established with the L4 proxy, which directs traffic through Hermes.\n\u003cimg src=\"https://heroku-blog-files.s3.amazonaws.com/posts/1698271115-IMG_2488.jpeg\" alt=\"IMG_2488\"\u003e\nWhen an \u003ca href=\"https://devcenter.heroku.com/articles/heroku-runtime-router-2-0#enable-router-2-0\"\u003eapp has Router 2.0 enabled\u003c/a\u003e, the L4 proxy instead funnels traffic over an in-memory listener to Router 2.0, then out to the app’s web dynos. Hermes is cut out of the network path.\n\u003cimg src=\"https://heroku-blog-files.s3.amazonaws.com/posts/1698339992-IMG_5679.jpeg\" alt=\"IMG_5679\"\u003e\nThis sort of architecture has a particular name — the “\u003ca href=\"https://www.redhat.com/architect/pros-and-cons-strangler-architecture-pattern\"\u003eStrangler pattern\u003c/a\u003e” — and it involves inserting a form of middleman between clients and the old system you want to replace. The middleman directs traffic, dividing it between the old system and a new system that is built out incrementally. The major advantage of such a setup is that “big bang” changes or “all-at-once” cut-overs are completely avoided. However, both the old and the new systems live on the same production hot path while the development of the new system is in progress. What has this meant for Router 2.0? Well, we had to lay a complete production-ready foundation early on.\u003c/p\u003e\n\u003ch2 class='anchored'\u003e\n  \u003ca name='living-on-the-hot-path' href='#living-on-the-hot-path'\u003eLiving on the Hot Path\u003c/a\u003e\n\u003c/h2\u003e\n\n\u003cp\u003eHeroku has always been an opinionated hosting and deployment platform that caters to general use cases. In our products, we optimize for stability while delivering innovation. Within the framing of Router 2.0, this commitment to stability meant we had to do a few things \u003cem\u003ebefore\u003c/em\u003e releasing beta.\u003c/p\u003e\n\u003ch3 class='anchored'\u003e\n  \u003ca name='automate-router-deployments' href='#automate-router-deployments'\u003eAutomate Router Deployments\u003c/a\u003e\n\u003c/h3\u003e\n\n\u003cp\u003eUp until recently, deploying Router 2.0 meant creating a new release and manually triggering router fleet cycles across all our production clouds. This process wasn’t only tedious and time-consuming, but it was also really error prone. We fixed this by building out an automation pipeline, outfitted with gates on availability metrics, performance metrics, and smoke tests. Anytime a router release fails on just one of these health indicators, it doesn’t advance to the next stage of deployment.\u003c/p\u003e\n\u003ch3 class='anchored'\u003e\n  \u003ca name='load-test-continuously' href='#load-test-continuously'\u003eLoad Test Continuously\u003c/a\u003e\n\u003c/h3\u003e\n\n\u003cp\u003eAn important aspect of vetting the new sharded environments in 2021 was load testing the L4 proxy/Hermes combo. At the time, this was a significant manual undertaking. After manually running these tests, it became obvious that we would need a more practical load testing story while developing Router 2.0. In response, we built a load testing system to continuously push our staging routers to their limits and trigger scaling policies, so that we can also validate our autoscaling setup. This framework has been immensely valuable for Router 2.0 development, catching bugs and regressions before they ever hit production. The results of these load tests feed right back into our deployment pipeline, blocking any deploys that don’t live up to our internal service level objectives.\u003c/p\u003e\n\u003ch3 class='anchored'\u003e\n  \u003ca name='introduce-network-error-logging' href='#introduce-network-error-logging'\u003eIntroduce Network Error Logging\u003c/a\u003e\n\u003c/h3\u003e\n\n\u003cp\u003eTraditionally, routing health has been measured through the use of “checkee” apps. These are web-server applications that we deploy across our production Common Runtime clouds and constantly probe from corresponding ”checker“ apps that run in Private Spaces. The checker-checkee duo allows us to mimic and measure our customers’ routing experience. In recent years, the gaps in this model have become more apparent. Namely, our checkees only represent the tiniest fraction of traffic pumping through the router at any given time. In addition, we can’t within our checkers possibly account for all the various client types and configurations that may be used to connect to the platform.\u003c/p\u003e\n\n\u003cp\u003eTo address the gap, we introduced \u003ca href=\"https://devcenter.heroku.com/changelog-items/2678\"\u003eNetwork Error Logging\u003c/a\u003e (NEL) to both Hermes and Router 2.0. It’s an experimental \u003ca href=\"https://www.w3.org/\"\u003eW3C\u003c/a\u003e standard that enables the measurement of routing layer performance by collecting real-time data about network failures from web browsers. Google Chrome, Microsoft Edge, and certain mobile clients already \u003ca href=\"https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/NEL#browser_compatibility\"\u003esupport\u003c/a\u003e the spec. NEL ensures our engineers maintain a more holistic understanding of the routing experience actually felt by clients.\u003c/p\u003e\n\u003ch2 class='anchored'\u003e\n  \u003ca name='the-future' href='#the-future'\u003eThe Future\u003c/a\u003e\n\u003c/h2\u003e\n\n\u003cp\u003eCompletely retiring Hermes will take time. We’re only at the end of the beginning of that journey. As detailed in the \u003ca href=\"https://devcenter.heroku.com/articles/heroku-runtime-router-2-0\"\u003eDev Center article\u003c/a\u003e, Router 2.0 isn’t complete yet because it doesn’t support the full list of features on our \u003ca href=\"https://devcenter.heroku.com/articles/http-routing\"\u003eHTTP Routing\u003c/a\u003e page. We’re working on it. We’ll soon be adding \u003ca href=\"https://github.com/heroku/roadmap/issues/34\"\u003eHTTP/2 support\u003c/a\u003e, one of the most requested features, to both the Common Runtime and Private Spaces. However, in the Common Runtime, HTTP/2 will only be available when your app is using Router 2.0.\u003c/p\u003e\n\n\u003cp\u003eOur aim is to achieve feature parity with Hermes, plus a little more, over the next few months. Once we’re there, we’ll focus on a migration plan that involves flagging apps into Router 2.0 automatically. Much like in the migration from classic environments to sharded environments, we’ll break the process out into phases based on small batches of apps in similar \u003ca href=\"https://devcenter.heroku.com/articles/dyno-types#dyno-tiers-and-mixing-dyno-types\"\u003edyno tiers\u003c/a\u003e. This approach gives us time to pause between phases and assess the performance of the new system.\u003c/p\u003e\n\u003ch2 class='anchored'\u003e\n  \u003ca name='participating' href='#participating'\u003eParticipating\u003c/a\u003e\n\u003c/h2\u003e\n\n\u003cp\u003eWe hope that you, our customers, can help us validate the new router well before it becomes the default. You can enable Router 2.0 for a Common Runtime app, by running:\u003c/p\u003e\n\n\u003cp\u003e\u003ccode\u003eheroku labs:enable http-routing-2-dot-0 -a \u0026lt;app\u0026gt;\u003c/code\u003e\u003c/p\u003e\n\n\u003cp\u003eIf you choose to enroll, you can submit feedback by commenting on the \u003ca href=\"https://github.com/heroku/roadmap/issues/219\"\u003eHeroku Public Roadmap item\u003c/a\u003e or \u003ca href=\"https://help.heroku.com/tickets/new?id=4\"\u003ecreating a support ticket\u003c/a\u003e.\u003c/p\u003e\n\u003ch2 class='anchored'\u003e\n  \u003ca name='conclusion' href='#conclusion'\u003eConclusion\u003c/a\u003e\n\u003c/h2\u003e\n\n\u003cp\u003eDelivering new features to a platform like Heroku is never as simple as flipping an on/off switch. When we deliver something to our customers, there’s always a mountain of behind-the-scenes effort put into it. Simply stated, we write a lot of software to ensure the software that you see works the way it should.\u003c/p\u003e\n\n\u003cp\u003eWe’re proud of the work we’ve done so far on Router 2.0, and we’re excited for what’s coming next. If you enroll your applications in the beta, keep an eye on the \u003ca href=\"https://devcenter.heroku.com/articles/heroku-runtime-router-2-0\"\u003eRouter 2.0 Dev Center\u003c/a\u003e page and the \u003ca href=\"https://devcenter.heroku.com/changelog\"\u003eHeroku Changelog\u003c/a\u003e. We’ll be posting updates about new features as they become available.\u003c/p\u003e\n\n\u003cp\u003eThanks for reading and happy coding!\u003c/p\u003e\n","published_at":"2023-10-30T17:00:00.000Z","permalink":"https://blog.heroku.com/router-2dot0-the-road-to-beta","tags":[],"summary":"As part of our commitment to infrastructure modernization, Heroku is upgrading the Common Runtime routing layer with the beta release of Router 2.0."},{"title":"More Predictable Shared Dyno Performance","content":"\u003cp\u003eIn this post, we’d like to share an example of the kind of behind-the-scenes work that the Heroku team does to continuously improve the platform based on customer feedback. \u003c/p\u003e\n\n\u003cp\u003eThe Heroku Common Runtime is one of the best parts of Heroku. It’s the modern embodiment of the principle of computing resource \u003ca href=\"https://en.wikipedia.org/wiki/Time-sharing\"\u003etime-sharing\u003c/a\u003e pioneered by John McCarthy and later by UNIX, which evolved into the underpinnings of much of modern-day cloud computing. Because Common Runtime resources are safely shared between customers, we can offer dynos very efficiently, participate in the \u003ca href=\"https://blog.heroku.com/github-student-developer-program\"\u003eGitHub Student Program\u003c/a\u003e, and run the \u003ca href=\"https://www.heroku.com/open-source-credit-program\"\u003eHeroku Open Source Credit Program\u003c/a\u003e.\u003c/p\u003e\n\n\u003cp\u003eWe previously allowed individual dynos to burst their CPU use relatively freely as long as capacity was available. This is in the spirit of time-sharing and improves overall resource utilization by allowing some dynos to burst while others are dormant or waiting on I/O.\u003c/p\u003e\n\n\u003cp\u003eLiberal bursting has worked well over the years and most customers got excellent CPU performance at a fair price. Some customers using shared dynos occasionally reported degraded performance, however, typically due to “noisy neighbors”: other dynos on the same instance that, because of misconfiguration or malice, used much more than their fair share of the shared resources. This would manifest as random spikes in request response times or even \u003ca href=\"https://devcenter.heroku.com/articles/addressing-h12-errors-request-timeouts\"\u003eH12 timeouts\u003c/a\u003e.\u003c/p\u003e\n\n\u003cp\u003eTo help address the problem of noisy neighbors, over the past year Heroku has quietly rolled out improved resource isolation for shared dyno types to ensure more stable and predictable access to CPU resources. Dynos can still burst CPU use, but not as much as before. While less flexible, this will mean fairer and more predictable access to the shared resources backing \u003ccode\u003eeco\u003c/code\u003e, \u003ccode\u003ebasic\u003c/code\u003e, \u003ccode\u003estandard-1X\u003c/code\u003e, and \u003ccode\u003estandard-2X\u003c/code\u003e Dynos. We’re not changing how many dynos run on each instance, we’re only ensuring more predictable and fair access to resources. Also note that Performance, Private, and Shield type Dynos are not affected because they run on dedicated instances.\u003c/p\u003e\n\n\u003cp\u003e\u003ci\u003eWant to see what we’re working on next or suggest improvements for Heroku? Check out our \u003ca href=\"https://github.com/heroku/roadmap\"\u003eroadmap on GitHub\u003c/a\u003e! Curious to learn about all the other recent enhancements we’ve made to Heroku? Check out the \u003ca href=\"https://blog.heroku.com/heroku-2022-roundup\"\u003e‘22 roundup\u003c/a\u003e and  \u003ca href=\"https://blog.heroku.com/heroku-feedback-news-2023-q1\"\u003eQ1 ’23 News\u003c/a\u003e blog posts.\u003c/i\u003e\u003c/p\u003e\n","published_at":"2023-04-11T18:00:05.143Z","permalink":"https://blog.heroku.com/more-predictable-shared-dyno-performance","tags":[],"summary":"In this post, we’d like to share an example of the kind of behind-the-scenes work that the Heroku team does to continuously improve the platform based..."},{"title":"The Adventures of Rendezvous in Heroku’s New Architecture","content":"\u003cp\u003e\u003cem\u003eThis article was originally authored by Srinath Ananthakrishnan, an engineer on the Heroku Runtime Networking Team\u003c/em\u003e \u003c/p\u003e\n\u003ch2 class='anchored'\u003e\n  \u003ca name='summary' href='#summary'\u003eSummary\u003c/a\u003e\n\u003c/h2\u003e\n\n\u003cp\u003eThis following story outlines a recent issue we saw with migrating one of our internal systems over to a new EC2 substrate and in the process breaking one of our customer’s use cases. We also outline how we went about discovering the root of the issue, how we fixed it, and how we enjoyed solving a complex problem that helped keep the Heroku customer experience as simple and straightforward as possible!\u003c/p\u003e\n\u003ch2 class='anchored'\u003e\n  \u003ca name='history' href='#history'\u003eHistory 📖\u003c/a\u003e\n\u003c/h2\u003e\n\n\u003cp\u003eHeroku has been leveraging AWS and EC2 since the very early days. All these years, the \u003ca href=\"https://devcenter.heroku.com/articles/dyno-runtime#common-runtime\"\u003eCommon Runtime\u003c/a\u003e has been running on \u003ca href=\"https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-classic-platform.html\"\u003eEC2 Classic\u003c/a\u003e and while there have always been talks about moving to the more performant and feature rich VPC architecture that AWS offers, we hadn’t had the time and personnel investment to make it a reality until very recently. The results of that effort were captured in a previous blog post titled \u003ca href=\"https://blog.heroku.com/faster-dynos-for-all\"\u003eFaster Dynos for All\u003c/a\u003e\u003c/p\u003e\n\n\u003cp\u003eWhile our Common Runtime contains many critical components, including our instance fleet to run app containers, our routers and several other control plane components, one of the often overlooked but yet critical components is Rendezvous, our bidirectional proxy server that enables \u003ca href=\"https://devcenter.heroku.com/articles/one-off-dynos\"\u003eHeroku Run\u003c/a\u003e sessions to containers. This is the component that lets customers run what are called one-off dynos that are used for a wide range of use-cases ranging from a simple prompt to execute/test a piece of code to complex CI scenarios.\u003c/p\u003e\n\u003ch2 class='anchored'\u003e\n  \u003ca name='architecture-of-rendezvous' href='#architecture-of-rendezvous'\u003eArchitecture of Rendezvous 💪\u003c/a\u003e\n\u003c/h2\u003e\n\n\u003cp\u003eRendezvous has been a single-instance server from time immemorial. It is a sub-200 line Ruby script that runs on an EC2 instance with an EIP attached to it. The ruby process receives TLS connections directly, performs TLS termination and proxies bidirectional connections that match a given hash.\u003c/p\u003e\n\n\u003cp\u003e\u003cimg src=\"https://heroku-blog-files.s3.amazonaws.com/posts/1642548509-Screen%20Shot%202021-12-07%20at%2011.16.39%20AM.png\" alt=\"Screen Shot 2021-12-07 at 11\"\u003e\u003c/p\u003e\n\n\u003cp\u003eEvery Heroku Run/One-off dyno invocation involves two parties - the client which is usually the Heroku CLI or custom implementations that use the \u003ca href=\"https://devcenter.heroku.com/articles/platform-api-reference\"\u003eHeroku API\u003c/a\u003e and the dyno on one of Heroku’s instances deep in the cloud. The existence of Rendezvous is necessitated by one of the painful yet essential warts of the Internet - NATs.\u003c/p\u003e\n\n\u003cp\u003eBoth the client and the dyno are behind NATs and there’s no means for them to talk to each other through these pesky devices. To combat this, the Heroku API returns an \u003ccode\u003eattach_url\u003c/code\u003e as part of the \u003ccode\u003ecreate_dyno\u003c/code\u003e \u003ca href=\"https://devcenter.heroku.com/articles/platform-api-reference#dyno-create\"\u003erequest\u003c/a\u003e which lets the client reach the dyno. The \u003ccode\u003eattach_url\u003c/code\u003e also contains a 64 bit hash to identify this specific session in Rendezvous. The same \u003ccode\u003eattach_url\u003c/code\u003e with the exact hash is passed on by our dyno management system to an agent that runs on our EC2 instance fleet which is responsible for the lifecycle of dynos.\u003c/p\u003e\n\n\u003cp\u003e\u003cimg src=\"https://heroku-blog-files.s3.amazonaws.com/posts/1642548531-Screen%20Shot%202021-12-07%20at%2011.57.32%20AM.png\" alt=\"Screen Shot 2021-12-07 at 11\"\u003e\u003c/p\u003e\n\n\u003cp\u003eOnce both the systems receive the \u003ccode\u003eattach_url\u003c/code\u003e with the same hash, they make a TLS request to the host, which is a specific instance of Rendezvous. Once the TLS session is established, both sides send the hash as the first message which lets Rendezvous identify which session the connection belongs to. Once the two sides of the session are established, Rendezvous splices them together, thus creating a bi-directional session between the CLI/user and the dyno.\u003c/p\u003e\n\u003ch2 class='anchored'\u003e\n  \u003ca name='a-unique-use-case-of-rendezvous' href='#a-unique-use-case-of-rendezvous'\u003eA unique use-case of Rendezvous 💡\u003c/a\u003e\n\u003c/h2\u003e\n\n\u003cp\u003eWhile the majority of customers use Rendezvous via \u003ccode\u003eheroku run\u003c/code\u003e commands executed via the CLI, some clients have more sophisticated ways of needing containers to be started arbitrarily via the \u003ca href=\"https://devcenter.heroku.com/articles/platform-api-reference#dyno-create\"\u003eHeroku API\u003c/a\u003e. These clients programmatically create a dyno via the API and also establish a session to the \u003ccode\u003eattach_url\u003c/code\u003e.\u003c/p\u003e\n\n\u003cp\u003eOne of our customers utilized Rendezvous in a very unique way by running an app in a Private Space that received client HTTP requests and within the context of a request, issued another request to the Heroku API and to Rendezvous. They had a requirement to support requests across multiple customers and to ensure isolation between them, they opted to run each of their individual customer’s requests inside \u003ca href=\"https://devcenter.heroku.com/articles/one-off-dynos#one-off-dynos\"\u003eone-off dynos\u003c/a\u003e. The tasks in the one-off dyno runs are expected to take a few seconds and were usually well within the expected maximum response time limit of \u003ca href=\"https://devcenter.heroku.com/articles/limits#http-timeouts\"\u003e30s by the Heroku router\u003c/a\u003e.\u003c/p\u003e\n\u003ch2 class='anchored'\u003e\n  \u003ca name='oh-something-s-broken' href='#oh-something-s-broken'\u003eOh! Something’s broken!\u003c/a\u003e\n\u003c/h2\u003e\n\n\u003cp\u003eIn July 2021, we moved Rendezvous into AWS VPCs as part of our effort to evacuate EC2 classic. We chose similar generation instances for this as our instance in classic. As part of this effort, we also wanted to remove a few of the architectural shortcomings of rendezvous - having a single EIP ingress and also manual certificate management for terminating TLS.\u003c/p\u003e\n\n\u003cp\u003e\u003cimg src=\"https://heroku-blog-files.s3.amazonaws.com/posts/1642548541-Screen%20Shot%202022-01-07%20at%201.36.18%20PM.png\" alt=\"Screen Shot 2022-01-07 at 1\"\u003e\u003c/p\u003e\n\n\u003cp\u003eBased on experience with other routing projects, we decided to leverage \u003ca href=\"https://docs.aws.amazon.com/elasticloadbalancing/latest/network/introduction.html\"\u003eNetwork Load Balancers\u003c/a\u003e that AWS offers. From a performance perspective, these were also significantly better - our internal tests revealed that NLBs offered 5-7x more throughput in comparison to the EIP approach. We also decided to leverage the NLB’s TLS termination capabilities which allowed us to stop managing our own certificate and private key manually and rely on AWS ACM to take care of renewals in the future.\u003c/p\u003e\n\n\u003cp\u003eWhile the move was largely a success and most customers didn’t notice this and their \u003ccode\u003eheroku run\u003c/code\u003e sessions continued to work after the transition, our unique customer immediately hit \u003ca href=\"https://devcenter.heroku.com/articles/error-codes#h12-request-timeout\"\u003eH12s\u003c/a\u003e on their app that spawns one-off dynos. Almost immediately, we identified this issue to Rendezvous sessions taking longer than the 30s limit imposed by the Heroku Router. We temporarily switched their app to use the classic path and sat down to investigate.\u003c/p\u003e\n\u003ch2 class='anchored'\u003e\n  \u003ca name='where-s-the-problem' href='#where-s-the-problem'\u003eWhere’s the problem! 😱\u003c/a\u003e\n\u003c/h2\u003e\n\n\u003cp\u003eOur first hunch was that the TLS termination on the NLB wasn’t happening as expected but our investigations revealed that TLS was appropriately terminated and the client was able to make progress following that. The next line of investigation was in Rendezvous itself. The new VPC-based instances were supposed to be faster, so the slowdown was something of a mystery. We even tried out an instance type that supported 100Gbps networking but the issue persisted. As part of this effort, we also had upgraded the Ruby version that Rendezvous was running on - and you guessed it right - we attempted a downgrade as well. This proved to be inconclusive as well.\u003c/p\u003e\n\n\u003cp\u003eAll along we also suspected this could possibly be a problem in the language runtime of the recipient of the connection, where the bytes were available in the userspace buffer of the runtime but the API call was not notified or there is a race condition. We attempted to mimic the data pattern between the client and the process in the one-off dyno by writing our own sample applications. We actually built sample applications in two different languages with very different runtimes. Both these ended up having the same issues in the new environment as well. 🤔\u003c/p\u003e\n\n\u003cp\u003eWe even briefly considered altering the Heroku Router’s timeout from 30s, but it largely felt like spinning a roulette wheel since we weren’t absolutely sure where the problem was.\u003c/p\u003e\n\u003ch2 class='anchored'\u003e\n  \u003ca name='nailing-it-down' href='#nailing-it-down'\u003eNailing it down! 🔍\u003c/a\u003e\n\u003c/h2\u003e\n\n\u003cp\u003eAs part of the troubleshooting effort, we also added some more logging on the agent that runs on every EC2 instance that is responsible for maintaining a connection with Rendezvous and the dyno. This agent negotiates TLS with Rendezvous and establishes a connection and sets up a \u003ccode\u003epty\u003c/code\u003e terminal connection on the dyno side and sets up stdin/stdout/stderr channels with the same. The client would send requests in a set-size byte chunks which would be streamed by this agent to the dyno. The same agent would also receive bytes from the dyno and stream it back to Rendezvous to send it back to the client. Through the logs on the agent, we determined that there were logs back and forth indicating traffic between the dyno and Rendezvous when connections worked. However, for the abnormal case, there were no logs indicating traffic coming from the dyno after a while and the last log was bytes being sent to the dyno.\u003c/p\u003e\n\n\u003cp\u003eDigging more, we identified two issues with this piece of code:\u003c/p\u003e\n\n\u003col\u003e\n\u003cli\u003eThis piece of code was single threaded - i.e. a single thread was performing an \u003ccode\u003eIO.select\u003c/code\u003e on the TCP socket on the Rendezvous side and the terminal reader on the dyno.\u003c/li\u003e\n\u003cli\u003eWhile #1 itself is not a terrible problem, it became a problem with the use of NLBs which are more performant and have different TLS frame characteristics.\u003c/li\u003e\n\u003c/ol\u003e\n\n\u003cp\u003e#2 meant that the NLB could potentially send much larger TLS frames than the classic setup where the Rendezvous ruby process would have performed TLS.\u003c/p\u003e\n\n\u003cp\u003eThe snippet of code that had the bug was as follows.\u003c/p\u003e\n\n\u003cpre\u003e\u003ccode\u003e# tcp_socket can be used with IO.select\n# ssl_socket is after openssl has its say\n# pty_reader and pty_writer are towards the dyno\ndef rendezvous_channel(tcp_socket, ssl_socket, pty_reader, pty_writer)\n    if o = IO.select([tcp_socket, pty_reader], nil, nil, IDLE_TIMEOUT)\n        if o.first.first == pty_reader\n\n            # read from the pty_reader and write to ssl_socket\n\n        elsif o.first.first == tcp_socket\n\n            # read from the ssl_socket and write to pty_writer\n\n        end\n    end\nend\n\u003c/code\u003e\u003c/pre\u003e\n\n\u003cp\u003eSince the majority of the bytes were from the client, this thread would have read from the \u003ccode\u003essl_socket\u003c/code\u003e and written them to the \u003ccode\u003epty_writer\u003c/code\u003e. With classic, these would have been small TLS frames which would mean that an \u003ccode\u003eIO.select\u003c/code\u003e readability notification would correspond to a single read from the SSL socket which would in-turn read from the TCP socket.\u003c/p\u003e\n\n\u003cp\u003eHowever, with the shards, the TLS frames from the NLB end up being larger, and a previous read from the \u003ccode\u003essl_socket\u003c/code\u003e could end up reading more bytes off of the \u003ccode\u003etcp_socket\u003c/code\u003e which could potentially block \u003ccode\u003eIO.select\u003c/code\u003e till the \u003ccode\u003eIDLE_TIMEOUT\u003c/code\u003e has passed. It’s not a problem if the \u003ccode\u003eIDLE_TIMEOUT\u003c/code\u003e is a relatively smaller number but since this was larger than the 30s limit imposed by the Heroku Router, \u003ccode\u003eIO.select\u003c/code\u003e blocking here resulted in that timer elapsing resulting in H12s.\u003c/p\u003e\n\n\u003cp\u003eIn fact, the \u003ca href=\"https://www.rubydoc.info/stdlib/core/IO.select\"\u003eRuby docs\u003c/a\u003e for \u003ccode\u003eIO.select\u003c/code\u003e specifically talk about this \u003ca href=\"https://www.rubydoc.info/stdlib/core/IO.select#:%7E:text=The%20most%20likely%20situation%20is%20that%20OpenSSL%3A%3ASSL%3A%3ASSLSocket%20buffers%20some%20data.%20IO.select%20doesn%27t%20see%20the%20buffer.%20So%20IO.select%20can%20block%20when%20OpenSSL%3A%3ASSL%3A%3ASSLSocket%23readpartial%20doesn%27t%20block.\"\u003eissue\u003c/a\u003e.\u003c/p\u003e\n\n\u003cblockquote\u003e\n\u003cp\u003eThe most likely situation is that OpenSSL::SSL::SSLSocket buffers some data. IO.select doesn\u0026#39;t see the buffer. So IO.select can block when OpenSSL::SSL::SSLSocket#readpartial doesn\u0026#39;t block.\u003c/p\u003e\n\u003c/blockquote\u003e\n\n\u003cp\u003eAccording to the Linux kernel on the instance, there were no bytes to be read from the \u003ccode\u003etcp_socket\u003c/code\u003e while there were still bytes being left to read from the buffers in openssl since we only read partially the last time around.\u003c/p\u003e\n\u003ch2 class='anchored'\u003e\n  \u003ca name='the-fix' href='#the-fix'\u003eThe fix 💜\u003c/a\u003e\n\u003c/h2\u003e\n\n\u003cp\u003eOnce we had identified the issue, it was rather straightforward for us to fix this. We made the code dual threaded - one each for one side of the connection and also fixed the way we read from the sockets and did an \u003ccode\u003eIO.select\u003c/code\u003e. With this code change, we ensured that we wouldn’t perennially block where there are bytes lying around to be read.\u003c/p\u003e\n\n\u003cp\u003eWe deployed this fix to our staging environments and after thorough testing we moved the customer over to the VPC-based rendezvous. The customer subsequently confirmed that the issue was resolved and all our remote offices erupted in roars of cheer after that. It was 🍰 time.\u003c/p\u003e\n\u003ch2 class='anchored'\u003e\n  \u003ca name='conclusion' href='#conclusion'\u003eConclusion\u003c/a\u003e\n\u003c/h2\u003e\n\n\u003cblockquote\u003e\n\u003cp\u003eComputers are fun, computers are hard!\u003c/p\u003e\n\n\u003cp\u003eTry to run a platform and you’ll often say, oh my god!\u003c/p\u003e\n\n\u003cp\u003eGratifying and inspiring it is, to run our stack\u003c/p\u003e\n\n\u003cp\u003eFor if you lose their trust, it’s hard to get it back ...\u003c/p\u003e\n\u003c/blockquote\u003e\n\n\u003cp\u003eRunning a platform makes you appreciate more of \u003ca href=\"https://www.hyrumslaw.com/\"\u003eHyrum’s Law\u003c/a\u003e, every day. Customers find interesting ways to use your platform and they sure do keep you on your toes to ensure you provide the best in class service. At Heroku we have always taken pride in our mission to make life easy for customers and we are grateful to have got the opportunity to demonstrate that yet again as part of this endeavor.\u003c/p\u003e\n\n\u003cp\u003eThanks are in order for all the folks who tirelessly worked on identifying this issue and fixing it. In alphabetical order - David Murray, Elizabeth Cox, Marcus Blankenship, Srinath Ananthakrishnan, Thomas Holmes, Tilman Holschuh and Will Farrington.\u003c/p\u003e\n","published_at":"2022-01-19T17:00:00.000Z","permalink":"https://blog.heroku.com/adventures-of-rendevous","tags":["tls","VPC","ec2","networking","NAT","ruby","ssl","AWS"],"summary":"A story of how we encountered a critical error for a customer use-case and the root cause analysis that followed. "},{"title":"Building a Monorepo with Yarn 2","content":"\u003cp\u003eIn true JavaScript fashion, there was no shortage of releases in the JavaScript ecosystem this year. This includes the Yarn project’s release of Yarn 2 with a compressed cache of JavaScript dependencies, including a Yarn binary to reference,  that can be used for a zero-install deployment. \u003c/p\u003e\n\n\u003cp\u003e\u003cimg src=\"https://heroku-blog-files.s3.amazonaws.com/posts/1608668413-yarn.png\" alt=\"Ball of yarn and knitting needles illustration\"\u003e\u003c/p\u003e\n\n\u003cp\u003eYarn is a package manager that also provides developers a project management toolset. Now, Yarn 2 is now officially supported by Heroku, and Heroku developers are able to take advantage of leveraging zero-installs during their Node.js builds. We’ll go over a popular use case for Yarn that is enhanced by Yarn 2: using workspaces to manage dependencies for your monorepo.\u003c/p\u003e\n\n\u003cp\u003eWe will cover taking advantage of Yarn 2’s cache to manage monorepo dependencies. Prerequisites for this include a development environment with Node installed. To follow these guides, set up an existing Node project that makes use of a \u003ccode\u003epackage.json\u003c/code\u003e too. If you don’t have one, use the \u003ca href=\"https://github.com/heroku/node-js-getting-started\"\u003eHeroku Getting Started with Node.js Project\u003c/a\u003e.\u003c/p\u003e\n\u003ch2 class='anchored'\u003e\n  \u003ca name='workspaces' href='#workspaces'\u003eWorkspaces\u003c/a\u003e\n\u003c/h2\u003e\n\n\u003cp\u003eFirst off, what are workspaces? Workspaces is Yarn’s solution to a monorepo structure for a JavaScript app or Node.js project. A monorepo refers to a project, in this case, a JavaScript project, that has more than one section of the code base. For example, you may have the following set up:\u003c/p\u003e\n\n\u003cpre\u003e\u003ccode class=\"language-javascript\"\u003e/app\n - package.json\n - /server\n   - package.json\n - /ui\n   - package.json\n\u003c/code\u003e\u003c/pre\u003e\n\n\u003cp\u003eYour JavaScript server has source code, but there’s an additional front end application that will be built and made available to users separately. This is a popular pattern for setting up a separation of concerns with a custom API client, a build or testing tool, or something else that may not have a place in the application logic. Each of the subdirectory’s \u003ccode\u003epackage.json\u003c/code\u003e will have their own dependencies. How can we manage them? How do we optimize caching? This is where Yarn workspaces comes in.\u003c/p\u003e\n\n\u003cp\u003eIn the root \u003ccode\u003epackage.json\u003c/code\u003e, set up the subdirectories under the \u003ccode\u003eworkspaces\u003c/code\u003e key. You should add this to your \u003ccode\u003epackage.json\u003c/code\u003e:\u003c/p\u003e\n\n\u003cpre\u003e\u003ccode class=\"language-javascript\"\u003e\u0026quot;workspaces\u0026quot;: [\n    \u0026quot;server\u0026quot;,\n    \u0026quot;ui\u0026quot;\n]\n\u003c/code\u003e\u003c/pre\u003e\n\n\u003cp\u003eFor more on workspaces, visit here: \u003ca href=\"https://yarnpkg.com/features/workspaces\"\u003ehttps://yarnpkg.com/features/workspaces\u003c/a\u003e\u003c/p\u003e\n\n\u003cp\u003eAdditionally, add the \u003ccode\u003eworkspaces-tools\u003c/code\u003e plugin. This will be useful when running workspace scripts that you’ll use later. You can do this by running:\u003c/p\u003e\n\n\u003cpre\u003e\u003ccode class=\"language-javascript\"\u003eyarn plugin import workspace-tools\n\u003c/code\u003e\u003c/pre\u003e\n\u003ch2 class='anchored'\u003e\n  \u003ca name='setting-up-yarn' href='#setting-up-yarn'\u003eSetting up Yarn\u003c/a\u003e\n\u003c/h2\u003e\n\n\u003cp\u003eIf you’re already using Yarn, you have a \u003ccode\u003eyarn.lock\u003c/code\u003e file already checked into your code base’s git repository. There’s other files and directories that you’ll need up to set up the cache. If you aren’t already using Yarn, install it globally.\u003c/p\u003e\n\n\u003cpre\u003e\u003ccode class=\"language-javascript\"\u003enpm install -g yarn\n\u003c/code\u003e\u003c/pre\u003e\n\n\u003cp\u003e\u003cem\u003eNote: If you don’t have Yarn \u0026gt;=1.22.10 installed on your computer, update it with the same install command.\u003c/em\u003e\u003c/p\u003e\n\n\u003cp\u003eNext, set up your Yarn version for this code base. One of the benefits of using Yarn 2 is that you’ll have a checked in Yarn binary that will be used by anyone that works on this code base and eliminates version conflicts between environments.\u003c/p\u003e\n\n\u003cpre\u003e\u003ccode class=\"language-javascript\"\u003eyarn set version berry\n\u003c/code\u003e\u003c/pre\u003e\n\n\u003cp\u003eA \u003ccode\u003e.yarn\u003c/code\u003e directory and \u003ccode\u003e.yarnrc.yml\u003c/code\u003e file will both be created that need to be checked into git. These are the files that will set up your project’s local Yarn instance.\u003c/p\u003e\n\u003ch2 class='anchored'\u003e\n  \u003ca name='setting-up-the-dependency-cache' href='#setting-up-the-dependency-cache'\u003eSetting Up the Dependency Cache\u003c/a\u003e\n\u003c/h2\u003e\n\n\u003cp\u003eOnce Yarn is set up, you can set up your cache. Run yarn install:\u003c/p\u003e\n\n\u003cpre\u003e\u003ccode class=\"language-javascript\"\u003eyarn\n\u003c/code\u003e\u003c/pre\u003e\n\n\u003cp\u003eBefore anything else, make sure to add the following to the \u003ccode\u003e.gitignore\u003c/code\u003e:\u003c/p\u003e\n\n\u003cpre\u003e\u003ccode class=\"language-javascript\"\u003e# Yarn\n.yarn/*\n!.yarn/cache\n!.yarn/releases\n!.yarn/plugins\n!.yarn/sdks\n!.yarn/versions\n\u003c/code\u003e\u003c/pre\u003e\n\n\u003cp\u003eThe files that are ignored will be machine specific, and the remaining files you’ll want to check in. If you run \u003ccode\u003egit status\u003c/code\u003e, you’ll see the following:\u003c/p\u003e\n\n\u003cpre\u003e\u003ccode class=\"language-javascript\"\u003eUntracked files:\n  (use \u0026quot;git add \u0026lt;file\u0026gt;...\u0026quot; to include in what will be committed)\n    .gitignore\n    .pnp.js\n    .yarn/cache/\n    yarn.lock\n\u003c/code\u003e\u003c/pre\u003e\n\n\u003cp\u003eYou’ve created new files that will speed up your install process:\u003c/p\u003e\n\n\u003cul\u003e\n\u003cli\u003e\u003ccode\u003e.pnp.js\u003c/code\u003e - This is the Plug’n’Play (PnP) file. The PnP file tells your Node app or build how to find the dependencies that are stored in \u003ccode\u003e.yarn/cache\u003c/code\u003e.\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003e.yarn/cache\u003c/code\u003e - This directory will have the dependencies that are needed to run and build your app.\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003eyarn.lock\u003c/code\u003e - The lock file still is used to lock the versions that are resolved from the \u003ccode\u003epackage.json\u003c/code\u003e.\u003c/li\u003e\n\u003c/ul\u003e\n\n\u003cp\u003eCheck all of this in to git, and you’re set. For more information about Yarn 2’s zero-install philosophy, read here: \u003ca href=\"https://yarnpkg.com/features/zero-installs\"\u003ehttps://yarnpkg.com/features/zero-installs\u003c/a\u003e\u003c/p\u003e\n\u003ch2 class='anchored'\u003e\n  \u003ca name='adding-dependencies-to-subdirectories' href='#adding-dependencies-to-subdirectories'\u003eAdding Dependencies to Subdirectories\u003c/a\u003e\n\u003c/h2\u003e\n\n\u003cp\u003eNow that Yarn and the cache are set up, we can start adding dependencies. As initially shown, we have a \u003ccode\u003eserver\u003c/code\u003e directory and a \u003ccode\u003eui\u003c/code\u003e directory. We can assume that each of these will be built and hosted differently. For example, my server is written in TypeScript, using Express.js for routing, and running on a Heroku web dyno. For the front end app, it is using Next.js. The build will be run during the app’s build process.\u003c/p\u003e\n\n\u003cp\u003eAdd \u003ccode\u003eexpress\u003c/code\u003e to the server \u003ccode\u003edependencies\u003c/code\u003e. \u003c/p\u003e\n\n\u003cpre\u003e\u003ccode class=\"language-javascript\"\u003eyarn workspace server add express\n\u003c/code\u003e\u003c/pre\u003e\n\n\u003cp\u003eAdditionally, add \u003ccode\u003e@types/express\u003c/code\u003e and \u003ccode\u003etypescript\u003c/code\u003e to the \u003ccode\u003edevDependencies\u003c/code\u003e. You can use the \u003ccode\u003e-D\u003c/code\u003e flag to indicate that you’re adding \u003ccode\u003edevDependencies\u003c/code\u003e. \u003c/p\u003e\n\n\u003cpre\u003e\u003ccode class=\"language-javascript\"\u003eyarn workspace server add @types/express typescript -D\n\u003c/code\u003e\u003c/pre\u003e\n\n\u003cp\u003eWe now have our dependencies in our \u003ccode\u003eserver\u003c/code\u003e workspace. We just need to create our \u003ccode\u003eui\u003c/code\u003e workspace. Next, build a Next.js app with the \u003ccode\u003eyarn create\u003c/code\u003e command.\u003c/p\u003e\n\n\u003cpre\u003e\u003ccode class=\"language-javascript\"\u003eyarn create next-app ui\n\u003c/code\u003e\u003c/pre\u003e\n\n\u003cp\u003eFinally, run \u003ccode\u003eyarn\u003c/code\u003e again to update the cache and check these changes into git.\u003c/p\u003e\n\u003ch2 class='anchored'\u003e\n  \u003ca name='running-scripts-with-workspaces' href='#running-scripts-with-workspaces'\u003eRunning Scripts with Workspaces\u003c/a\u003e\n\u003c/h2\u003e\n\n\u003cp\u003eThe last piece is to run scripts within the workspaces. If you look through your source code, you’ll see that there’s one global cache for all dependencies under your app’s root directory. Run the following to see all the compressed dependencies:\u003c/p\u003e\n\n\u003cpre\u003e\u003ccode class=\"language-javascript\"\u003els .yarn/cache\n\u003c/code\u003e\u003c/pre\u003e\n\n\u003cp\u003eNow, lets run build scripts with workspaces. First, set up the workspace. For server, use \u003ccode\u003etsc\u003c/code\u003e to build the TypeScript app. You’ll need to set up a TypeScript config and a \u003ccode\u003e.ts\u003c/code\u003e file first:\u003c/p\u003e\n\n\u003cpre\u003e\u003ccode class=\"language-javascript\"\u003ecd server\nyarn dlx --package typescript tsc --init\ntouch index.ts\n\u003c/code\u003e\u003c/pre\u003e\n\n\u003cp\u003e\u003ccode\u003eyarn dlx\u003c/code\u003e will run a command from a package so that it doesn’t need to be installed globally. It’s useful for one-off initializing commands, like initializing a TypeScript app.\u003c/p\u003e\n\n\u003cp\u003eNext, add the build step to the \u003ccode\u003eserver/package.json\u003c/code\u003e.\u003c/p\u003e\n\n\u003cpre\u003e\u003ccode class=\"language-javascript\"\u003e\u0026quot;scripts\u0026quot;: {\n    \u0026quot;build\u0026quot;: \u0026quot;tsc\u0026quot;,\n    \u0026quot;start\u0026quot;: \u0026quot;node index.js\u0026quot;\n},\n\u003c/code\u003e\u003c/pre\u003e\n\n\u003cp\u003eChange directories back to the application level, and run the build.\u003c/p\u003e\n\n\u003cpre\u003e\u003ccode class=\"language-javascript\"\u003ecd ..\nyarn workspace server build\n\u003c/code\u003e\u003c/pre\u003e\n\n\u003cp\u003eYou’ll see that a \u003ccode\u003eserver/index.js\u003c/code\u003e file is created. Add \u003ccode\u003eserver/*.js\u003c/code\u003e to the \u003ccode\u003e.gitignore\u003c/code\u003e.\u003c/p\u003e\n\n\u003cp\u003eSince we already have \u003ccode\u003ebuild\u003c/code\u003e and \u003ccode\u003estart\u003c/code\u003e scripts in our Next.js app (created by the \u003ccode\u003eyarn create\u003c/code\u003e command), add a build script at the root level \u003ccode\u003epackage.json\u003c/code\u003e.\u003c/p\u003e\n\n\u003cpre\u003e\u003ccode class=\"language-javascript\"\u003e\u0026quot;scripts\u0026quot;: {\n    \u0026quot;build\u0026quot;: \u0026quot;yarn workspaces foreach run build\u0026quot;\n},\n\u003c/code\u003e\u003c/pre\u003e\n\n\u003cp\u003eThis is when the \u003ccode\u003eworkspaces-tool\u003c/code\u003e plugin is used. Run \u003ccode\u003eyarn build\u003c/code\u003e from your app’s root, and both of your workspaces will build. Open a second terminal, and you’ll be able to run \u003ccode\u003eyarn workspace server start\u003c/code\u003e and \u003ccode\u003eyarn workspace ui start\u003c/code\u003e in each terminal and run the Express and Next servers in parallel.\u003c/p\u003e\n\u003ch2 class='anchored'\u003e\n  \u003ca name='deploy-to-heroku' href='#deploy-to-heroku'\u003eDeploy to Heroku\u003c/a\u003e\n\u003c/h2\u003e\n\n\u003cp\u003eFinally, we can deploy our code to Heroku. Since Heroku will run the script is in the \u003ccode\u003epackage.json\u003c/code\u003e under \u003ccode\u003estart\u003c/code\u003e, add a script to the \u003ccode\u003epackage.json\u003c/code\u003e.\u003c/p\u003e\n\n\u003cpre\u003e\u003ccode class=\"language-javascript\"\u003e\u0026quot;scripts\u0026quot;: {\n    \u0026quot;build\u0026quot;: \u0026quot;yarn workspaces foreach run build\u0026quot;,\n    \u0026quot;start\u0026quot;: \u0026quot;yarn workspaces server start\u0026quot;\n},\n\u003c/code\u003e\u003c/pre\u003e\n\n\u003cp\u003e\u003ca href=\"https://devcenter.heroku.com/articles/deploying-nodejs#specifying-a-start-script\"\u003eHeroku will use the \u003ccode\u003estart\u003c/code\u003e script\u003c/a\u003e from the \u003ccode\u003epackage.json\u003c/code\u003e to start the \u003ccode\u003eweb\u003c/code\u003e process on your app.\u003c/p\u003e\n\u003ch2 class='anchored'\u003e\n  \u003ca name='conclusion' href='#conclusion'\u003eConclusion\u003c/a\u003e\n\u003c/h2\u003e\n\n\u003cp\u003eThere are \u003ca href=\"https://yarnpkg.com/features\"\u003eplenty more features\u003c/a\u003e that Yarn, and specifically Yarn 2, offers that are useful for Heroku developers. Check out the Yarn docs to see if there are additional workspace features that may work nicely with Heroku integration. As always, if you have any feedback or issues, please \u003ca href=\"https://github.com/heroku/heroku-buildpack-nodejs/issues/new/choose\"\u003eopen an Issue on GitHub\u003c/a\u003e.\u003c/p\u003e\n","published_at":"2020-12-22T20:53:08.379Z","permalink":"https://blog.heroku.com/building-a-monorepo-with-yarn-2","tags":["javascript","yarn","node","monorepo"],"summary":"Yarn 2 is officially supported by Heroku. Here's a popular use case for Yarn enhanced by Yarn 2: using workspaces to manage dependencies for your monorepo."}]