On Software Testing

The goal of this post is to provide a practical, yet high-level introduction into software testing. I hope to establish a shared understanding of different types of software tests, and how to incorporate them into software development workflows. However, it is not within the scope of this post to compare different philosophies of software testing. Furthermore, this is not a “how-to” or step-by-step guide on how to write tests. Where applicable, I have provided references to more technical guides on how to specifically write a piece of software test.

Why testing software?

Testing is a core tenet of stable product delivery and high-throughput engineering teams. By covering core functionality with robust testing, software teams can afford to iterate quickly while keeping the risk of regressions and outages low.

Testing is not, however, a way to ignore good practices in developing software. Defensive programming, comprehensive error handling, documentation, code reviews, alerting, etc., cannot be deemphasized as a result of having high test coverage.

Testing Strategy

The diagram below shows the hierarchy of test types. The various types of tests should be run at intervals proportionate to their priority and position on the pyramid, i.e., unit tests run the most frequently, followed by integration tests, while UI tests will be run the least. These test runs should be built into the lifecycle hooks of the project through code version systems and CI/CD platforms. The following sections explain different test types in more detail.

Unit tests

Unit testing focuses on individual pieces of code; aka the “Unit”. Unit tests should ensure that the functions inside a given Unit, given any reasonable state we think could occur, should all work and return values as expected.

Because individual software units require limited functionality to run, you can run unit tests as you incrementally develop the codebase. The small scope of unit tests makes it easier to pinpoint where the problems are. Unit tests are the fastest to write, maintain, and automate. They only require modifications if their associated unit has changed directly. It is easier to predict and codify all the possible states of a particular unit, compared to a large component.

In short, automated unit tests should be the most comprehensive coverage and form the basis of our testing pyramid.

How to use unit tests

• Use automation tools such as pre-commit hooks to incorporate unit tests into the local development process.
• Run unit tests on a production-like environment on the CI server to ensure the build works in higher environments.
• Unit tests should run on most CI events, including a local commit, a commit push to a remote branch, opening a PR, release, etc.

Integration tests

Integration tests combine the individual software modules and test them together as a group. They test whether more than one component (unit, service, endpoint, database, client, API, etc.) are compatible and can work together. Integration tests are more important in microservice architectures where different services are developed independently and by different developers.

How to use integration tests

• Keep the scope of the integration tests to only what is required. Integration tests usually take longer to run, compared to unit tests. As a result, they can be more disruptive to the development process.
• Use integration tests when you cannot reliably mock a request or response in a unit test.
• Most integration tests only need to be run on a remote CI server. But, developers should also have access to a local environment for running integration tests before committing code to the remote branch.
• The decision to run which integration tests locally versus on the CI server should be taken on a case-by-case basis, based on how long does it take for the integration test runs and feasibility of creating a local environment that is close enough to the production environment.

UI / API tests

UI / API tests are sanity checks that ensure the most critical paths have not incurred any major regressions across deployments. They help catch potential issues that are hard to replicate in the CI/CD pipeline – e.g., changes in networking, routing, authentication, third-party APIs.

Automated UI tests are time-consuming to write and maintain, and test failures are harder to pinpoint. As a result, UI and API tests should be relied upon only as a last line of defense in comparison to Integration tests and unit tests.

How to use UI tests

• When possible, UI tests should be automated using tools such as Selenium and Cypress.
• Manual UI tests should only be used as a safety check, and for “acceptance testing”.
• UI tests should be run on only critical deployments (staging and prod). They should not be automatically run before every commit, push, or PR event.
• Before a staging release, UI tests should be run against a staging-like environment. This is usually possible with proper set up of CI/CD pipelines.
• Before a production release, UI tests should be run against a production-like environment.
• Breaking UI tests should roll back the release.
• The “heartbeat” version of the automated UI test will run the UI test suite against the production environment periodically, and generate an alarm (Slack, Rollbar, PagerDuty) if a test fails.

UI test type	What does it test	When is it run	What happens if fails
Automated	App functionality and responsiveness	During CI process (only for higher-env releases)	Stop / rollback the release
Heartbeat	App functionality and responsiveness	Periodically, on a schedule (e.g., every 15min)	Generate alerts and reports
Manual	App functionality, and product intended features	During a release event	Stop / rollback the release

A Phased Approach to Improving Test Coverage

For teams that are getting started with software testing, a phased approach is suggested to make a smooth transition to the desired test coverage.

Phase 1: Awareness and enablement

The goal of this phase is to bring awareness about test coverage, and enable the team to start writing good unit tests. This sends the signal to the team that test coverage will be prioritized in the organization. In particular:

• Connect the CI pipeline to a code coverage tool to capture and share code coverage metrics.
• Provide resources such as testing best practices and share testing libraries to improve skill sets and consistency across the team.
• Invest in writing shared libraries that could be used by the team members to write tests.

Phase 2: Enforce test coverage guidelines

The goal of this phase is to enforce desired outcomes and behaviors. This phase is about building a culture of testing. During this phase:

• Make sure all unit tests and selected integration tests are run in the CI pipeline.
• Enforce CI guides on PR merge:
• All unit and integration tests must pass (no exceptions).
• New PRs should not degrade test coverage by more than a small threshold.

Phase 3: Integrate tests into development lifecycle

The goal of this phase is to make testing a natural and integrated part of development. This phase is about creating a habit.

• Git’s pre-push hooks provide a place to run unit tests locally before pushing code to a remote branch. This ensures even the feature branches pass the tests during development.
• Optionally, integration tests can be run at this stage as well if they are sufficiently fast (this will depend on the codebase, test runner, etc., so should be taken on a case-by-case basis).

Phase 4: Continuous improvement of existing codebase

The goal of this phase is to pay down “test tech debt”. This phase is about kaizen (continuous improvement) of the codebase.

• Based on the ongoing investigation and error discovery, identify areas of the existing code that could benefit from higher test coverage.
• Dedicate a percentage of the team’s time to paying down “test tech debt”.

Phase 5: Beyond reactive testing

The goal of this step is to look beyond testing as a side-process of software development, and more as one of the main activities of the product development cycle. This phase is about codifying stable software development into the DNA of the team.

• Engage QA engineers from the early stages of product definition and development.
• Generate Requirements Traceability Matrix (RTM) in coordination with the product team. 
• Generate a Testing & QA Plan and communicate with the team. The Testing plan should at least include the following components:
  • Define the scope of testing.
  • Specify test cases. This should be done in coordination with the RTM. Also include test classification (e.g., functional vs non-functional).
  • Specify the testing strategy, including which features or components require integration tests, automated UI tests, manual UI tests, etc. 
  • Define testing objectives.
  • Specify testing tools and platforms.
  • A playbook for how to respond to testing outcomes. 
• Clarify roles and responsibilities during release events. The QA team needs to be empowered to stop a release or call for a rollback, if the release does not meet the QA requirements.