by Jake Worth on November 8, 2016

Gold Master Testing is a technique for evaluating complex legacy systems.

The general idea is that you take a known input, such as a database, run it through a function that changes the data, and then compare the output with an approved version of the output.

It's ideal for systems that are mature, where little change in the output is expected. And it's ideal for systems that are complex and difficult to test.

This week my pair and I wrote a test like this, and I wanted to share the experience. Our work developed through four general phases: preparation, testing, evaluation, and maintenance.

The first step we took was to acquire a production database dump and restore it into a local Postgres database.

Once we had the database dump, we wrote a small Rake task to transform it into plaintext SQL.

namespace :gold do
  desc "DATABASE will be used to generate the new gold test database"
  task :update_db do
    db_name = ENV.fetch('DATABASE')
    destination = Rails.root.join('spec/fixtures/gold_master.sql')
    sh "pg_dump #{db_name} --attribute-inserts --column-inserts --no-tablespaces --disable-triggers --data-only > #{destination}"
  end
end

Some noteworthy flags affecting our pg_dump.

• --attribute-inserts and --column-inserts to add explicit column names
• --disable-triggers to temporarily disable triggers on the target tables while the data is reloaded
• --data-only to dump only the data, not the schema

We chose to use Ruby's sh because it echoes the command before running it.

With a valid data dump, we were ready to test.

The focal point of the test is the Transformer.transform function. Here's the RSpec test that covers it:

describe 'gold master' do
  it 'produces a consistent result' do

    ActiveRecord::Base.connection.execute <<-SQL
      truncate schema_migrations;
      #{Rails.root.join('spec/fixtures/gold_master.sql').read}
    SQL

    actual = Transformer.transform

    gold_master_file = Rails.root.join('spec/fixtures/gold_master.txt')
    gold_master = gold_master_file.read

    if gold_master != actual
      gold_master_file.write(actual)
    end

    expect(actual).to eq(gold_master)
  end
end

Breaking this down, we start with a database transaction, which truncates the schema table and then executes the plaintext SQL statements. The result is our production data dumped into the test database.

Next we call Transformer.transform and assign the output to the variable actual.

Then, we read an approved version of the output, generated on a previous run. If it doesn't match the Gold Master, we write the file with the same name. Writing the file adds unstaged changes to Git, which has some interesting benefits we will look at in the next step.

Finally, we make our assertion— if the two outputs don't match, the test fails.

Using the Database Cleaner transaction strategy was an crucial part of making this all work. The production database can only exist in the scope of the single test we care about.

On a mature system, this test should pass most of the time. If it fails, we broke accepted behavior. Like any good test, it is a safeguard against that happening, but on an exacting scale.

If it does fail, we've written the file, so our Git log has unstaged changes to the Gold Master we must address.

If the changes are desired, great; we can commit them; if not, we can abandon them.

Like any test, this one is only as good as the maintenance that supports it. It is more brittle than the usual test by its nature. Disciplined upkeep would include running all migrations on the local database and restoring a fresh production dump from time to time. Data changes shouldn't affect the result because we are testing behavior, not data.

This was a fun test to write. Thank you to Brian Dunn, who paired with me through the implementation.

If you have a mature, complex application, consider Gold Master Testing. With the proper preparation, testing, evaluation, and maintenance, it can be a valuable addition to a robust test suite.

^{Image Source: https://commons.wikimedia.org/wiki/File:400-oz-Gold-Bars-AB-01.jpg}