Web application development agency

Fixing low code coverage when using test helper with Minitest, SimpleCov and rake

Introduction

You might face low code coverage in non-Rails environment ¹ (for example, when using Sinatra) when running rake test task, whereas running single tests via ruby -I test test/unit/some_test.rb works fine. This article explains why it happens and how to fix it.

The article assumes you have already configured SimpleCov, set up the Minitest Rake test task, and that you have a test helper file at test/test_helper.rb.

Problem analysis

Most suggestions found online recommend placing and invoking SimpleCov at the very top of your test helper (the entry point for all tests). However, if you’ve followed the documentation, it should already be there — meaning the issue lies elsewhere.

To debug this, we first need to understand what happens when we run rake test. Minitest provides a useful rake test:cmd task, which outputs the command that the test suite will use to run the tests. Let’s execute it and examine the output:

$ rake test:cmd
ruby -Ilib:test:. -w -e 'require "minitest/autorun"; require "test/unit/one_test.rb"; require "test/unit/another_test.rb"; require "test/test_helper.rb";
# lots of other test files...'

Can you immediately spot what’s wrong? If you run the task multiple times, you’ll notice that:

• Test files are loaded in random order.
• More importantly, they typically load after the test helper (which initializes SimpleCov).

Minitest’s test runner treats the test helper and regular test files absolutely the same, resulting in non-deterministic (random) load order.

The Minitest::TestTask class has a @framework instance variable that holds the initialization code executed when the test suite starts. By default, this code runs the test suite by requiring minitest/autorun, which triggers Minitest.autorun. This reveals the root issue: Because Minitest initializes first, SimpleCov — which is initialized in the test helper — cannot properly instrument the test suite for coverage analysis.

Simply placing SimpleCov.start at the beginning of your test helper is not sufficient: The test helper itself must be executed before the regular tests when the test suite runs. The next section will discuss this in detail.

The cure

To fix the issue, follow the next steps:

1. Tune the creation of the test task in the Rakefile as follows:

    Minitest::TestTask.create do |t|
      t.framework = %(require "test/test_helper.rb")
      t.test_globs = ["test/**/*_test.rb"]
    end
   

   First, we override Minitest’s default initialization to explicitly load test_helper.rb before any other test files. Then we modify the test file pattern to exclude the test helper itself, since the default pattern (test/**/test_*.rb) would otherwise include it twice — once through our explicit load and once through the automatic test file discovery.

2. Add require "minitest/autorun" to test_helper.rb after the SimpleCov setup. This loads and initializes Minitest, which is necessary because we overrode the default test framework startup code in the @framework variable. Without this, Minitest won’t execute any tests.

After running rake test:cmd again, you should see test_helper.rb always loaded first, followed by the remaining test files:

$ rake test:cmd
ruby -Ilib:test:. -w -e 'require "test/test_helper.rb"; require "test/unit/one_test.rb"; require "test/unit/another_test.rb"; # lots of other test files...'

That’s it!

Published: October 24, 2024

Updated: April 7, 2025

Tags: ruby, minitest, simplecov, rake, code coverage, test helper

https://github.com/rails/rails/issues/27088#issuecomment-261385679.