by Mary Lee on February 14, 2019

In a recent Rails project, we were tasked with the migration of a group of ActiveRecord models to a microservice that had not yet been built.

This raised an interesting question for us: how do we migrate data out of an application when we don’t have anything to replace it with?

The originally proposed solution was to build a layer between the controllers and models that would delegate out to the ActiveRecord models for now, and then when the new microservice was done, would delegate out to an ActiveResource object. This solution raised some concerns for us.

• How would we update the models and jobs that interacted with the models being extracted, when the layer was only present in the controllers?
• How do we know the performance impact of moving from a database record to fetching data from an API?
• How do we know we’ve properly converted everything when we’re ultimately still using the same ActiveRecord object?
• Could we write useful tests when we weren’t sure what the final objects were going to look like?
• What would be the proof of concept? When would we know that this approach was going to work?

We had also noticed that the work already done to build out the middle layer in the controllers introduced a number of extra queries, which was a major performance concern for us.

We decided to do some brainstorming, and came up with a different approach: what if we created an internal API using the ActiveRecord models, and then build out and use API resource objects in the main application? We could build a JSON API and use JSON API resources to replace the ActiveRecord resources, moving them behind a namespace in the main application. This addressed a number of our concerns, and had some added benefits as well.

• Because we were fully replacing the ActiveRecord resources, we wouldn’t be running the risk of missed translations or changed behaviors with the new resources; if the JSON API resources didn’t support a method, they would throw an error.
• We could convert the models and jobs, along with the controllers
• We could identify and address performance impacts early on
• We could write meaningful tests, setting up the WebMock stubs that the application could use even after the switch to the microservice was made
• Once the microservice was complete, all that would be required to make the change to using the microservice would be to update the url for the JSON API resources
• By building out the API internally, we would be defining what the final API would need to implement, once the microservice started being built
• As a proof of concept, we could start converting just one of the models being migrated, and make sure that our approach was feasible

We chose two libraries to make the internal JSON API:

• json_api_client for creating the new API resources that would be replacing the existing ActiveRecord resources
• jsonapi-resources for creating the internal JSON API endpoints

We then started by creating some new namespaces in the controllers and models. We then moved the ActiveRecord model to the namespace, replacing it with a JsonApiClient::Resource, resulting in something like this:

• app/controllers/foo_controller.rb would exist as it originally did, but would be updated to use the new JsonApiClient::Resource
• app/controllers/local_api/foo_controller.rb would be a JSONAPI::ResourceController backed by the original ActiveRecord model
• app/models/foo.rb would be the JsonApiClient::Resource that would be used to replace the ActiveRecord resource throughout the application
• app/models/local_api/foo.rb would be the original ActiveRecord resource
• app/resources/local_api/foo_resource.rb would be the JSONAPI::Resource object that would pull from ActiveRecord to create the internal API

The namespacing also happened in the routes file:

resources :foo 

namespace :local_api do
  jsonapi_resources :foo
end

We made sure to maintain the same namespacing throughout the app as we made our changes, because our ultimate goal was to remove everything within those namespaces once the microservice was in use. Separating them out early on would make the final removal more straightforward.

For the purposes of development and debugging, we wanted to be able to run two servers on our Rails application, one for the main app, and one for the internal API. This would allow us to see all the calls being made to the API, and help us identify where we could optimize. We accomplished this by setting the JsonApiClient::Resources to look at localhost on a different port than the main app, and then starting two servers:

1. The main app server: bundle exec rails s
2. The API server: bundle exec rails s -p 3001 -P tmp/api_server.pid

With the API server running on a different port, we could update our JsonApiClient resources to use port 3001, thus seeing API requests in a separate log from the main application requests.

A final step in our setup was to include a helper method in the base API controller that would print out the API response when a debug flag was present.

module LocalAPI
  class BaseController < JSONAPI::ResourceController
    after_action do
      pp JSON.parse(response.body) if Rails.env.development? && ENV["API_DEBUG"]
    end
  end
end

Now if we ever wanted to see what the API was returning, we could start the API server with API_DEBUG=true bundle exec rails s -p 3001 -P tmp/api_server.pid, and it would print out the JSON response for the requests we were making.

Along the way in our development, we came across a few speed bumps, namely:

• Resolving pagination issues
• Handling large filter requests
• Custom sorts
• Polymorphism

Check out the rest of this series!

• Part 2: Solving Pagination
• Part 3: Handling Filtering
• Part 4: Custom Sorting
• Part 5: Polymorphism
• Part 6: Mimicking ActiveRecord
• Part 7: Testing