Creating Heroku Review Apps

I’ve been setting up Review Apps on Heroku to streamline our product process. Review apps let you automatically spin up new app intance for every github pull request, creating isolated testing environments for each feature. Pretty neat!

…in theory. In practice I hit some hurdles. For example, our app relies on shared third party resources and a large postgres data set. In this post I’ll explain how we handle elasticsearch and postgres in our review apps at Ascent.

Elasticsearch

Setup

We’re using a hosted elasticsearch provider, elastic.co. Our indices aren’t huge, but some of them are relatively slow to build. For example, we index the contents of tens of thousands of PDF documents. We want to copy our elasticsearch indices from our staging app into our review apps so we don’t have to wait hours for the indices to populate and the review apps to work properly.

Our provider doesn’t currently allow us to programmatically create new clusters, so all our staging and review app indices are in a single cluster. That means we have to configure each model to use an environment-specific index, rather than changing the base elasticsearch URL. Review apps provide an environment variable called HEROKU_APP_NAME which we use to create a suffix for the elasticsearch indices.

# create a global variable in config/initializers/elasticsearch.rb
ELASTICSEARCH_INDEX_SUFFIX = ''
if ENV['HEROKU_APP_NAME'].to_s.include? "-pr-"
  ELASTICSEARCH_INDEX_SUFFIX = "-#{ENV['HEROKU_APP_NAME']}"
end

# explicitly set the index name in any model using elasticsearch
class MyModel < ActiveRecord::Base
  include Elasticsearch::Model
  index_name "#{self.name.pluralize.underscore}#{ELASTICSEARCH_INDEX_SUFFIX}"
end

Now that we’re set up to use different indices in different environments, we have to actually create and populate those indices when the review app is created. Heroku has us covered: postdeploy scripts in app.json.

{
	"scripts": {
		"postdeploy": "bundle exec rake copy_elasticsearch_indices_to_review_app"
	}
}

I hit a couple wrinkles with the rake task itself, but Elasticsearch::Model does the heavy lifting.

# load all the models so they're available to be detected
(ActiveRecord::Base.connection.tables - %w[schema_migrations]).each do |table|
  table.classify.constantize rescue nil
end

ActiveRecord::Base.descendants.select { |model| model.included_modules.include?(Elasticsearch::Model) }.each do |model| 
  Rails.logger.info "Copying ES index for #{model.name}..."
  Elasticsearch::Model.client.reindex({body: { 
  	
    # The staging app uses the default index name from Elasticsearch::Model
    source: { index: model.name.pluralize.underscore }, 

    # The review app uses our suffixed index name
  	dest: { index: model.index_name }
  }})
end

To check whether you’ve done it right, find your elasticsearch URL. It’s probably in your heroku app config, personally I use heroku config | grep ELASTICSEARCH to grab it. Paste that base URL into your browser, and add the path /_cat/indices?v to see all your indices. As your review apps are being created, you should see your original set of unsuffixed indices as well as a new set with a -pr-123 suffix for each new review app.

Teardown

All these review app indices are taking up space on the cluster. When the review app is no longer active, we can delete the indices to free up space. Heroku again has us covered, though pr-predestroy scripts in app.json are not mentioned in the normal app.json docs.

# app.json
"scripts": {
  "pr-predestroy": "bundle exec rake delete_review_app_indices"
}

# elasticsearch.rake
task delete_review_app_indices: :environment do
  # ensure we don't accidentally delete indices we care about
  return unless Rails.env.staging?
  return unless ENV['HEROKU_APP_NAME'].present?

  Elasticsearch::Model.client.indices.delete({index: "*#{ENV['HEROKU_APP_NAME']}"})
end

Note the star in the Elasticsearch::Model arguments. The star is a wildcard, allowing us to match and delete any index suffixed with our app name.

Postgres

Setup

We’re fundamentally a data company, so many of our features revolve around handling domain-specific data sets. That means our test environments also need quite a bit of data in them. Unfortunately, Heroku only lets you add the most basic addon plans to review apps through your app.json config. Even though our staging app has a standard-0 postgres database, all review apps are created with hobby-dev databases. After a few unsuccessful approaches here’s where we finally ended up.

1. Add Heroku’s platform-api gem to the Gemfile and bundle:

   # Gemfile
   gem 'platform-api', '~> 2.1'
   

   ``

2. Instantiate an API client. Don’t prefix your API key variable with HEROKU_, they’ve reserved that namespace for themselves.

   # config/initializers/heroku.rb
   HEROKU_API_CLIENT = PlatformAPI.connect(ENV['API_KEY_FOR_HEROKU'])
   

   ``

3. Build a rake task that creates a new standard-0 database and makes it the primary database for your app. I had a really hard time googling how to programmatically promote a heroku postgres database to be the primary database for an app, eventually I found the answers in the Heroku CLI pg:promote code itself.

task create_standard_0_database: :environment do
  
  # Find the name of your hobby tier database.
  # This will be one of the autogenerated names from
  # Heroku, like postgresql-pointer-899183
  current = HEROKU_API_CLIENT.addon_attachment
                             .list_by_app(ENV['HEROKU_APP_NAME'])
                             .detect{|addon| addon['name'] == "DATABASE"}
  current_name = current['addon']['name']

  # The current add-on occupying the DATABASE attachment has no
  # other attachments. In order to promote this database without
  # error, we can create a secondary attachment.
  body = {
    app: { name: ENV['HEROKU_APP_NAME'] },
    addon: { name: current_name },
    namespace: nil,
    confirm: ENV['HEROKU_APP_NAME']
  }
  HEROKU_API_CLIENT.addon_attachment.create(body)

  # Create a new standard-0 database attached to the review app
  new_leader = HEROKU_API_CLIENT.addon.create(
  	ENV['HEROKU_APP_NAME'], 
  	{plan: "heroku-postgresql:standard-0"}
  )
  new_leader_id = new_leader['id']
  new_leader_name = new_leader['name']

  # wait for it to be provisioned
  while HEROKU_API_CLIENT.addon.info(new_leader_id)['state'] == 'provisioning'
    sleep 1
  end

  # Creating a new attachment for DATABASE effectively promotes this
  # database to be the primary database for the app.
  body = {
    name: "DATABASE",
    app: { name: ENV['HEROKU_APP_NAME'] },
    addon: { name: new_leader_name },
    namespace: nil,
    confirm: ENV['HEROKU_APP_NAME']
  }
  HEROKU_API_CLIENT.addon_attachment.create(body)

  # Copy the data from the staging environment to the review app
  staging_pg_url = HEROKU_API_CLIENT.config_var.info_for_app(ENV['HEROKU_PARENT_APP_NAME'])['DATABASE_URL']
  review_pg_url = HEROKU_API_CLIENT.config_var.info_for_app(ENV['HEROKU_APP_NAME'])['DATABASE_URL']

  system "pg_dump --no-owner #{staging_pg_url} | psql #{review_pg_url}"

end

4. Run this task in your postdeploy script:

# app.json
"scripts": {
  "postdeploy": "bundle exec rake create_standard_0_database && 
  					bundle exec rake db:migrate"
}

5. Leave the hobby-dev database alone. If we don’t have heroku-postgresql in the app.json addons, the rails app won’t boot and we don’t even get to the postdeploy script.

Teardown

Since all the postgres databases are addons to the app, they’ll be automatically destroyed when the review app is destroyed.

Workflow Tips

• When you’re trying to get review apps up and running, you can prototype most things in your local console.
• …which is good, because review apps take a pretty long time to build.
• You can manually create and destroy review apps in the Heroku dashboard.
• Use the EXCON_DEBUG=1 environment variable to get much more verbose error messages from the Heroku API.
• You can set EXCON_DEBUG in app.json if you can’t reproduce a Heroku API issue locally.