Supporting Large Data: Part 1

As apps have matured on Heroku, data sets have gotten much larger. Taps is designed to help development by providing a fast and easy way to transfer databases between local environments and Heroku. Today we launched taps 0.3 with a reworked architecture and a new set of features focused on large data sets:

  1. Push/Pull Specific Tables
    You can now choose which tables to push and pull. Specify a regex and taps will only push or pull the tables that match. To only pull specific tables, specify a comma delimited list. For example, to pull the logs and tags tables, run this command:
    heroku db:pull --tables logs,tags
  2. Resume Transfers
    Interruptions can happen when moving large datasets. Now when you interrupt (Ctrl-C) or an error occurs, taps will dump a session file that can be used to resume the transfer:
    heroku db:pull --resume-filename session_file.dat
  3. More Robust Schema Translation
    DB operations are now 100% powered by Sequel, making taps capable of handling more varied data and schemas.

  4. Improved Transfer Speed
    Taps now uses primary keys to move large data sets with constant transfer speed regardless of table size. Taps will fall back to ‘paging’ the data (OFFSET + LIMIT) if no primary key is available.

Run “sudo gem update heroku taps” to get the new version.

Browse the blog archives or subscribe to the full-text feed.