Intro and Scope

Code coverage refers to how much of your code is being executed while your automated tests are running. This metric is calculated by special tools that add tracing calls inside the binaries of your code. These special tools generate code coverage reports that can be used to find what parts of your code are covered (and not covered) by your tests. They also provide a percentage coverage as a metric to monitor the extent of code coverage.

CodeCov is an online tool to visualize and track unit test coverage on codebases. It uses the code coverage reports, and visualizes them in a helpful way for the development teams to review and monitor. CodeCov also has robust integrations with some CI/CD tools. This enables integrating unit test coverage monitoring within the CI/CD pipelines.

This document goes over integrating CodeCov into Github via Github Actions.

Step 1: Connect CodeCov and Github (admins only)

Goals:

Sign up for Codecov
Integrate desired Github repositories to CodeCov

Note (for developers): your organization admin will most likely perform these steps. If CodeCov is already integrated with your Github repos, jump to step 2 below.

Note (for admins): CodeCov has two ways to integrate with Github:

As a third-party app, authenticating via OAuth integration (legacy method). This method uses an upload token and team bot. The upload token to be added to the CI process, to enable OAuth authentication for codecov.
As a Github Marketplace App (native app, no need for an upload token or team bot). This option gives more granular control over which repositories CodeCov has access to, from within Github. This installation method does not require an upload token, since the app integration takes care of the authentication. The Github App is the recommended method to integrate CodeCov with Github.

Integration as a third-party app (OAuth integration)

This installation method is driven from codecov.io web app, meaning you initiate the integration and choose which repos to monitor from codecov application.
First, go to https://app.codecov.io/gh and log in with your Github credentials. If adding for an organization, you first would need to grant CodeCov with access to your Github Organization.
From top-right, select “Not yet setup” to see a list of your Github repos.

Click on “Set up repo”. This takes you to another page with instructions on how to integrate CodeCov into your CI process. The most important piece of information is the “upload token”. Fetch that and store it in a secure place.

Go to the desired Github repository, and add the upload token as a Secret.

CODECOV_TOKEN=….

Integration as a Github App

Go to https://github.com/apps/codecov
Click on “Configure”

Grant access to the CodeCov App for your org (may need an administrator to do this step).

Configure which repositories CodeCov should have access to.

Step 2: Add code coverage generate and upload steps to Github Actions

Goals:

Generate a code coverage report
Upload the coverage report to Codecov

Generating a code coverage report

As long as you have tests in your app and your coverage tool of choice prints data in a format compatible with Codecov, you can plug Codecov into your projects and see coverage metrics for whatever workflow you’re using. Here are the list of code coverage report tools by language:

For instance, in Python, the unit test library pytest-cov can be used to generate code coverage reports by using the following command:

pytest --cov=./

In Github Actions, add the following step to your workflow:

    - name: Generate coverage report
      run: |
        pip install pytest
        pip install pytest-cov
        pytest --cov=./ --cov-report=xml

Uploading the report to CodeCov

Once you generate the report, you need to upload it to CodeCov to visualize it on CodeCov app. The upload can be done using the Codecov Uploader. This usually happens during the CI/CD process.

CodeCov has a special uploader to upload the coverage report to the CI/CD platform. The new uploader is written in NodeJS and resolves some of the issues with the previous bash script. For the CI/CD platforms that CodeCov has direct integration, CodeCov publishes wrappers that abstract away the uploader.

When using CodeCov with one of the supported CI/CD platforms, use one of the dedicated wrappers for CodeCov uploader. These wrappers use native scripts for the CI/CD platform, and abstract away the details of the upload. CodeCov provides ecosystem-specific wrappers around the CodeCov Uploader in the following CI platforms:

Github Actions: Marketplace page, codebase
CircleCI
BitRise

If using the Github Actions wrapper, all you need to do is to add the following step to your GH Actions workflow:

    - name: Upload coverage to Codecov
      uses: codecov/codecov-action@v3
      with:
        token: ${{ secrets.CODECOV_TOKEN }}
        directory: ./coverage/reports/
        env_vars: OS,PYTHON
        fail_ci_if_error: true
        files: ./.coverage,./.coverage.xml ./coverage,./coverage.xml
        flags: unittests
        name: codecov-umbrella
        path_to_write_report: ./coverage/codecov_report.txt
        verbose: true

Note: if you have installed CodeCov as a third-party app, you will need to add the upload token as a secret to your CI/CD platform (e.g., Github Secrets).

When you create the pull request, the CI action will be run, which will generate and upload your coverage report to CodeCov.

See this link for the most up-to-date Github Action workflow sample, as suggested by CodeCov.

Step 3: Add CodeCov report as a comment to Pull Requests

Goals:

Add test coverage report as a comment to PRs..

All you need to do is add a codecov.yml file in the root of your project.

coverage:
  status:
    project:
      default:
        target: auto
        threshold: 0.5%
    patch: off

Step 4: Add CodeCov check to PR checks

Goals:

Enforce maintaining certain level of code coverage with the addition of each PR.

CodeCov can be integrated as a check in the Github PR process. You can specify a threshold to enforce new code will not degrade test coverage more than a certain amount. As long as the new code will not reduce the test coverage by more than that percentage, the CodeCov check will pass.

Add this in the Branch Protections Rules in the Github repo: