In case you’ve forgotten, the second declaration of the Agile Manifesto states a preference for:
Working Software Delivered Over Comprehensive Documentation
With this in mind, it should be no surprise that there is such an emphasis on testing in projects being built with Agile methodologies.
While it’s important to have testing, and to automate as much of it as possible, it’s also important to create tests with a strategy in mind. One great strategy to apply is Mike Cohn’s Agile Testing Pyramid.
Mike Cohn’s Agile Testing Pyramid
Mike’s Agile Testing Pyramid provides a great strategy for the types of tests to create and automate. It also provides guidance on how many of the different types of tests you should have.
Let’s take a brief look at the types of tests that make up the pyramid. Once we’ve done that, we’ll circle back and dig more deeply into each type of test, why it’s important, and how to write it.
The Foundation: Unit Tests
The foundation of Mike’s Agile Testing Pyramid is unit tests. Perhaps seventy percent of all your tests should be unit tests. I say perhaps to ensure you will feel free to flex a bit one way or the other.
The important thing is that most of your tests are unit tests. Shoot for 100% coverage with these, but ensure you’re at least in the high 90s.
The Middle: Service Tests
Mike defines a service as a business operation. In his example of a calculator, he suggests that a service would be an operation like add or multiply. If you’re using Clean Architecture, this would align with a single Interactor or Story.
Aim to have at least one service test for each service you’re testing, and make sure each one of these is an subcutaneous end-to-end test.
The Top: UI Tests
The top of Mike’s Agile Testing Pyramid are UI tests. Where the service tests are subcutaneous, these comes straight through the UI, using Selenium or something similar.
If you think about it, it’s likely that the same UI drives several service operations. Since these tests are sanity checks on the wiring between the UI and services, you should need far fewer of these.
Unit Tests – A Foundation to Build On
As development proceeds, unit tests give a developer confidence that they are building the thing right. Each build confirming that things work as the developer intended.
Because unit tests are applied directly to a single class – disconnected from its collaborators by installing test doubles – there should be one test for each outcome a class can produce. Because these tests have checks for each value that is emitted during each outcome we call them direct tests.
The result? When something breaks, you have a specific test method that fails. You can tell exactly what went wrong. Discovery time is reduced, and – instead of guessing – a developer can put their energy into fixing things.
This feedback enables a developer to move quickly, with high confidence, knowing that the code they are building works.
Service Tests – Proof That Everyone Gets Along
Unit tests are a great start, but they are only one piece of a good automated testing strategy. You also need to automate tests that prove the units work well together, and that’s where service tests come into play.
Service tests are subcutaneous, meaning they come in “under the skin” (the UI) and operate the entire stitched-together system. They do this by calling the controllers directly.
A lot of people want to keep the database out of these tests – just as with the unit tests. However, it’s really important to know that all the pieces of the system are working properly, so I’d probably have a specific test database instance that is pre-initialized with any static data and that has all dynamic or transactional data removed between test executions.
Otherwise, you’re not really testing the whole system.
UI Tests – Validation of UI Integration
Once service tests have executed, the only things left unknown are whether the UI can talk to the services, and whether the UI acts appropriately given certain outcomes (rosy vs exceptional path). If you consider that several error responses result in the same error behavior, you probably don’t need to check the error path for everything.
The same is probably true for the rosy path.
UI tests are generally the most fragile and the most expensive. Err on the light side in the start. You can always add more if you feel they are warranted.
Wrapping It Up
If you think about the relationship between these different types of tests, there are some things you can know that might not be obvious. When unit tests fail, the cause is clear. However, once we get to service tests – and especially once we get to UI tests – the cause may become more murky.
For UI and service test failures, it is possible that the error relates to that specific integration step (UI code for UI tests and service/component code for service tests) or that the problem is a result of the developer implementing the next level “down” incorrectly.
For example, a failing UI test may come about because a developer has incorrectly implemented the UI to service integration. However, failing this, it could also result from someone having stitched the service together incorrectly.
Just because a test passes doesn’t mean the code it tests is implemented properly. It simply means that it meets the developer’s expectation. For this reason, if you feel you should omit the UI tests, you should make sure that you do not skip the service tests.
Otherwise, the only integration testing you have is manual execution. While it certain can work, it will be much less efficient than an automated test. Particularly when you consider that you can factor defects back into tests and know that you are avoiding regressions.
Remember that, no matter how comprehensive your test suite is, they were written by humans. Humans documented the requirements that were used to implement the test and code too.
Generally, be somewhat suspect of your unit tests until you have integration tests that prove the units were written properly. And, realize that this does not diminish the value of the unit test. It simply means that humans are imperfect and computers do what we tell them to.
That’s it for this week! Until next time, this is Eddie Bush with Craftsmanship Counts reminding you to keep your tests passing and your code clean!