Merging Multiple SimpleCov Coverage Results

As part of our Roadmap opens a new window service at FastRuby.io opens a new window , we analyze the test suite of the application we are auditing to give a proper estimate on how long it will take us to upgrade. One of the tools we use for this is SimpleCov opens a new window .

Often times our clients use parallelization in their continuous integration tools. SimpleCov generates multiple .resultset.json files for the same codebase. However, our goal is to have a single .resultset.json for the whole application. In this blog post we are going to show you how we solved the problem.

Some of the applications we upgrade are outdated and setting them up can be difficult. Sometimes they have no documentation, missing steps on how to set them up, or outdated installation steps. Even after setup, the test suite of these applications can take several hours to complete and generate a coverage report.

We recently decided to take a different approach for this problem. Instead of executing the tests locally and generating the report, we rely on our client’s CI configuration to get the coverage data. The setup for this may be a little different for each type of CI, but in general it is pretty easy to configure the CI to give us the resultset.json.

We did however also find a problem with this approach. Continuous integration services (like CircleCI opens a new window ) allow you to use parallelization for any command and a common pattern is to execute different parts of your test suite in different containers.

This will make the tests run faster. The problem with this is that if you are running SimpleCov opens a new window it will generate a result for each of your containers. To have the full coverage report you’ll have to merge all results to generate one final coverage result.

This means that you will need to download each file individually. Once you have all of the files downloaded you can run a task to merge the results into one file.

The latest versions of SimpleCov, >= 0.18.0 have a collate method. Example script for a merging a directory with files(simplecov-resultset/.resultset_1.json):

namespace :coverage do

  task :report do
    require 'simplecov'

    SimpleCov.collate Dir["simplecov-resultset/.resultset_*.json"]
  end
end

If you happen to be using an older version of SimpleCov < 0.18.0 you can use this script that we wrote.

class SimpleCovMerger
  def self.report_coverage(base_dir:, ci_project_path:, project_path:)
    new(base_dir: base_dir, ci_project_path: ci_project_path, project_path: project_path).merge_results
  end

  attr_reader :base_dir, :ci_project_path, :project_path

  def initialize(base_dir:, ci_project_path:, project_path:)
    @base_dir = base_dir
    @ci_project_path = ci_project_path
    @project_path = project_path
  end

  def merge_results
    require "simplecov"
    require "json"

    results = []
    resultsets.each do |file|
      hash_result = JSON.parse(clean(File.read(file)))

      hash_result.each do |command_name, data|
        result = SimpleCov::Result.from_hash(command_name => data)
        results << result
      end
    end

    result = SimpleCov::ResultMerger.merge_results(*results)

    SimpleCov::ResultMerger.store_result(result)

    result.format!
  end

  private

  def resultsets
    Dir["#{base_dir}/.resultset-*.json"]
  end

  def clean(results)
    results.gsub(ci_project_path, project_path)
  end
end

To use it you’ll have to be aware of a couple of parameters:

• base_dir - This is the directory where you stored all your .resultset.json from your different containers/machines from your CI service
• ci_project_path - The path where your project is stored in your CI service
• project_path - The path of the project you are generating a coverage report

SimpleCovMerger.report_coverage(base_dir: "./resultsets", ci_project_path: "/home/ubuntu/the_project/", project_path: "/Users/bronzdoc/projects/fastruby/the_project/")

Conclusion

We hope you’ll find these ways of merging the SimpleCov results helpful for your purposes. Please let us know if you have a better way or any ideas for improving it. If you need help with your upgrade you can get in touch opens a new window .