How does testing fit into Continuous Delivery?
Continuous Delivery in practice means creating a delivery pipeline for your software. Pipeline stages build the software and perform increasingly stringent tests on it. As the software passes through each stage our confidence in the quality of the build increases. Eventually it reaches the end of the pipeline where it becomes a ‘Release Candidate” - a build that is judged to be of sufficient quality to be deployed into production.
If you build and test new versions of your software several times a day your testing will need to include a lot of automation to keep pace. Pipeline stages usually include automated unit, acceptance, and performance tests, but some testing is still done by hand. Once we have a build that passes all the automated tests it becomes available for self-service deployment to other environments for manual validation.
The business case for automation
The goal of automated testing in the Continuous Delivery Pipeline is to give us early feedback on software quality and ensure known problems don’t reappear. Research has shown that using DevOps practices, like test automation, leads to higher IT performance and improved business outcomes. Investing in automated testing means lower change failure rate for your software and improved delivery lead times.
A Continuous Delivery pipeline that includes extensive automated tests allows us to develop better ways of working. Giving developers rapid feedback on the effects of their changes improves their productivity and liberates them from repetitive manual work. An automated CD pipeline can also transform your ability to react to changes in the marketplace by enabling Lean Product Management practices such as hypothesis-driven development and A/B testing. Developers and testers can spend more of their time learning about customer needs and innovating.
How do I get started with Continuous Delivery and test automation?
The Test Automation Pyramid is a good place to start. Discovering defects as early as possible minimizes the cost of correcting them. Having a large number of very fast unit tests is also relatively cheap to maintain. These tests are a great start but they won’t discover all the problems. At the top of the pyramid we have automated end-to-end tests that exercise the system in the same way as an actual user. These tests are usually slow to run and expensive to maintain, so we will have only a few of them. In between the extremes of unit tests and end-to-end tests there are a lot of different kinds of tests you can include, like component tests, integration tests, contract tests, and load tests. Exactly what you should have will depend on your architecture.
The usual way to organize tests in your CD pipeline is to start with the fast, cheap tests and proceed to the slower, more expensive tests as you gain confidence in the quality of the code changes. If we discover defects later in the pipeline it should prompt us to make improvements to the testing earlier in the pipeline. That can involve writing new tests, or moving tests from a later to an earlier position in the pipeline. Even if these tests are expensive to run it can save us a lot of effort if we discover defects earlier. The aim is always to provide timely, valuable feedback on the effects of changes.
At Praqma we have helped several companies to improve their testing strategy as they move towards Continuous Delivery. We offer expert advice, strategy workshops, and coaching hands-on with your development teams. Find out how to improve your testing strategy by getting in touch with one of our experts today.
We help customers in Denmark, Sweden and Norway, and have offices in Copenhagen, Odense, Aarhus, Oslo, Malmö, Stockholm and Gothenburg.