Gold Master Testing
Gold Master Testing is a technique for evaluating complex legacy systems.
The general idea is that you take a known input, such as a database, run it through a function that changes the data, and then compare the output with an approved version of the output.
It's ideal for systems that are mature, where little change in the output is expected. And it's ideal for systems that are complex and difficult to test.
This week my pair and I wrote a test like this, and I wanted to share the experience. Our work developed through four general phases: preparation, testing, evaluation, and maintenance.
1. Preparation
The first step we took was to acquire a production database dump and restore it into a local Postgres database.
Once we had the database dump, we wrote a small Rake task to transform it into plaintext SQL.
namespace :gold do
desc "DATABASE will be used to generate the new gold test database"
task :update_db do
db_name = ENV.fetch('DATABASE')
destination = Rails.root.join('spec/fixtures/gold_master.sql')
sh "pg_dump #{db_name} --attribute-inserts --column-inserts --no-tablespaces --disable-triggers --data-only > #{destination}"
end
end
Some noteworthy flags affecting our pg_dump
.
--attribute-inserts
and--column-inserts
to add explicit column names--disable-triggers
to temporarily disable triggers on the target tables while the data is reloaded--data-only
to dump only the data, not the schema
We chose to use Ruby's sh
because it echoes the command before running it.
With a valid data dump, we were ready to test.
2. Testing
The focal point of the test is the Transformer.transform
function. Here's the RSpec test that covers it:
describe 'gold master' do
it 'produces a consistent result' do
ActiveRecord::Base.connection.execute <<-SQL
truncate schema_migrations;
#{Rails.root.join('spec/fixtures/gold_master.sql').read}
SQL
actual = Transformer.transform
gold_master_file = Rails.root.join('spec/fixtures/gold_master.txt')
gold_master = gold_master_file.read
if gold_master != actual
gold_master_file.write(actual)
end
expect(actual).to eq(gold_master)
end
end
Breaking this down, we start with a database transaction, which truncates the schema table and then executes the plaintext SQL statements. The result is our production data dumped into the test database.
Next we call Transformer.transform
and assign the output
to the variable actual
.
Then, we read an approved version of the output, generated on a previous run. If it doesn't match the Gold Master, we write the file with the same name. Writing the file adds unstaged changes to Git, which has some interesting benefits we will look at in the next step.
Finally, we make our assertion— if the two outputs don't match, the test fails.
Using the Database Cleaner transaction strategy was an crucial part of making this all work. The production database can only exist in the scope of the single test we care about.
3. Evaluation
On a mature system, this test should pass most of the time. If it fails, we broke accepted behavior. Like any good test, it is a safeguard against that happening, but on an exacting scale.
If it does fail, we've written the file, so our Git log has unstaged changes to the Gold Master we must address.
If the changes are desired, great; we can commit them; if not, we can abandon them.
4. Maintenance
Like any test, this one is only as good as the maintenance that supports it. It is more brittle than the usual test by its nature. Disciplined upkeep would include running all migrations on the local database and restoring a fresh production dump from time to time. Data changes shouldn't affect the result because we are testing behavior, not data.
Conclusion
This was a fun test to write. Thank you to Brian Dunn, who paired with me through the implementation.
If you have a mature, complex application, consider Gold Master Testing. With the proper preparation, testing, evaluation, and maintenance, it can be a valuable addition to a robust test suite.
Image Source: https://commons.wikimedia.org/wiki/File:400-oz-Gold-Bars-AB-01.jpg