Understanding Agile Testing Methodology

What do we mean by agile testing?

Testing is an integral part of software development along with coding, operations, understanding customer needs, and much more. We like models that represent a continuous or holistic approach to testing. The development process represents an endless loop, continually confirming — which is how we really develop software. We learn about functionality that our customers want, we build and deliver it, and then we learn how customers actually use it.

Agile Tests Principles

These principles still apply today to anyone on an agile team who wants to deliver the highest quality product possible.

• Provide ongoing feedback;
• Deliver value to the customer;
• Enable face-to-face communication;
• Have courage;
• Keep it simple;
• Practice continuous improvement;
• Respond to change;
• Auto-organize;
• Focus on people;
• To enjoy.

Testing Manifesto

The “test manifest” was created by Karen Greaves and Samantha Laing. Its manifesto reflects the change of required mindset for a successful agile testing approach.
We test throughout the development process, focus on the prevention
of bugs, we test much more than the functionality, and all
the team takes responsibility for quality.

Agile test definition

Collaborative testing practices that occur continuously, from inception to delivery and beyond, supporting the frequent delivery of value to our customers. Testing activities focus on building quality into the product, using rapid feedback loops to validate our understanding.
Practices strengthen and support the idea of team-wide responsibility for quality.

Test Planning in Agile Contexts

Testing at multiple levels requires extra planning. The release cycles
usually start with determining what can be delivered in the first “learning release”. Perhaps the only part of the functionality is delivered.

Features are divided into stories and prioritized so the team knows which to deliver first. It’s important for the team to understand the big picture before including stories in an iteration. When developers work on a story, they are more focused on making sure individual tasks are completed.
Testers sometimes fall into the trap of only thinking about the story they’re testing, so reminders of the big picture are important.

The image below shows the importance of a testing approach that includes all teams working towards a single product release. To give an overall picture of test coverage, consider bringing people from different teams together to create a test mind map or feature testing matrix (details in More Agile Testing) that spans the product.

Planning the Regression Tests

Regression testing is about making sure the system does what it did yesterday. Contemporary practices for delivering small changes to production often leave no time for full manual regression checking, so test automation is created as the product is developed.

Automated regression testing allows us to be confident in our product with quick feedback. Many (if not all) tests are run as part of continuous integration (CI). Some teams schedule slower tests to run less frequently. For example, running them several times a day instead of every build. Using feature flags in releases to “hide” changes from production users allows teams to do some testing activities asynchronously as they are continually deployed to production. The feature is “On” in production when all tests are complete.

Guiding Development with Examples

Concrete examples of desired and undesired system behavior help teams build a shared understanding of each feature and story. This allows them to build the right thing with fewer story rejects and shorter cycles between early development and production deployment. Testers contribute by asking for these concrete examples and using them to create executable tests that guide development. They can be the voice of experience in leading these conversations.

The Agile Testing Quadrants

The agile testing quadrants are a taxonomy of different types of testing. We use them as a thinking tool to help teams discuss what testing activities they might need and ensure they have the right people, resources, and environments to carry them out.

• Quadrant 1 (Q1): Technology-driven testing that guides development
• Quadrant 2 (Q2): Business-oriented testing that drives development
• Quadrant 3 (Q3): Business-oriented tests that criticize the product
• Quadrant 4 (Q4): Technology-oriented tests that criticize the product

The agile testing quadrants model helps teams think through the testing activities needed to give confidence to the product they are building. It also helps build a common test language across the team and the organization if used to help communicate across teams.

Quadrant 1

Teams can also use quadrants to talk about which tests to automate. Q1 tests are typically automated by developers writing production code. Many teams practice test-driven development (TDD), writing a small unit test for some small feature, then the code to make that test pass. Q1 tests are designed to run quickly as they test a small area of code and generally do not include interaction with other application layers or databases. They provide teams with the quick feedback they need to make code changes quickly and fearlessly.

Quadrant 2

The business-driven testing that drives development in Q2 is an important foundation for most teams. Product owners, developers, testers and others meet frequently to plan features and stories. They can use techniques such as example mapping to get business rules for each story, along with the examples that illustrate them. Teams that practice behavior-driven development (BDD), acceptance test-driven development (ATDD), or specification-by-example (SBE) turn these into scenarios that specify behavior as executable tests. These scenarios can be automated as code is written.

Quadrant 3

Testing in Q3 tends to be human-centric, finding out whether stories and features deliver the intended value for customers. Automation can be used to facilitate this testing through data or state configuration. It is becoming common for teams to do exploratory testing in production, using release feature flags to “hide” new functionality from customers until testing is complete. Q3 testing includes forms of production testing such as monitoring to understand what actually happens and how customers use features. Information learned from Q3 tests feeds back into Q2, often resulting in the creation of new stories or features.

Quadrant 4

Many Q4 tests rely on automation and tools, but some may require additional testing. For example, automated tools for accessibility (often abbreviated to “a11y”, representing the beginning and ending letters and the number of letters between them in the word “Accessibility”) are not yet as effective as manual exploratory testing for these functionalities. The results of these tests often feed back into Q1 activities as the team changes the code design to improve various quality attributes. Monitoring performance and errors in production, or testing for recoverability, can also be considered a type of test that falls under Q4. Today’s technology has made it feasible and safe to test in production. This doesn’t mean that we let customers find bugs for us, but that we learn for sure how the code behaves in a true production environment.

The Test Automation Pyramid

Mike Cohn’s Test Motors Pyramid has helped many times since the early 2000’s. We’ve tweaked it and resisted it ever since to make our intent clear, including the bubble cloud at the top to represent that not all regression tests can be instructed. Sometimes we need human-centric tests, which include exploratory tests (ET).

The test automation pyramid helps us think of ways to “push the tests down” by maximizing regression tests that are isolated to one part of an application and minimizing those that involve multiple parts of the system.

The classic pyramid does not mean that there is a certain number or percentage of automation tests at each level.

Seb’s model makes it clear that what makes a test more expensive is the number of application layers it requires to run. For example, it’s possible to have a unit level test of the UI that doesn’t involve any other layers of the application. You can use TDD for each isolated layer of the application, be it the server, the API, the UI or a microservice.

Teams that have automated tests can draw their pyramid “shape” to visualize where their current tests fit. Many teams start primarily with UI testing and an “upside down” or “ice cream cone” pyramid. Others may have an hourglass. Neither of these ways is necessarily wrong, but if current automation doesn’t meet the team’s needs, visuals can help you figure out what changes are needed.

Agile Testing Mindset

Testers are no longer the “quality police”, driving “will/don’t” decisions. Testers or team members who are performing testing activities can explain the risks and impacts of test results so that the company can make an informed decision about releasing to production.

As a team member with an agile testing mindset, this means that you are curious and want to learn more about everything to help you do your job. This means that you apply agile principles and values. It means collaborating with members of the technical and business team, keeping the big picture in mind when piecing together small increments of functionality. You’re focused on preventing bugs, so you don’t have to spend a lot of time finding bugs later.

Essential Practices

There are essential practices that have proven effective in helping teams build quality into their products.

1. Every team needs continuous integration (CI) to successfully deliver software at a frequent cadence over time. Each time a team member makes a change to the source code repository, they must initiate a build process that integrates all code changes and verifies them with automated tests. Each team has a deployment pipeline to create a release candidate and deploy it to a test or production environment — even if it includes manual stages.
2. Many teams still struggle to have reliable test environments that are as close to production as possible and allow you to easily control which version is deployed. Today’s cloud infrastructure offers even more options for temporary test environments to test a specific version and automatically deploy new versions to permanent test environments.
3. Programming and testing are part of the process. This is at the heart of the whole-team approach to testing and quality. Testing and programming happen together, hand in hand, from the idea of the feature to the evaluation of the feature in production.
4. The synergy between practices comes from doing all these core practices together. Test-driven development, collective code ownership, and continuous integration ensure consistency and fast feedback. Refactoring relies on automated regression testing. Agile practices are tried and true and are designed to be done together.

Conclusion

Every agile software delivery team has its own development journey.
apprenticeship. Our aim is to continually improve our ability
to deliver value to our customers on a regular basis, maintaining the standard desired quality for our business.

Every team is doing it with a unique combination of the business domain, software product, technology stack, structures and practices.
Over the years, we’ve found that among all these differences, certain
ingredients for success benefit all teams.

Factors such as testing being part of the development process, the agile testing mentality well established within the team, the automation of tests with a special focus on regression tests, and providing and obtaining feedback, among others, as mentioned above, help the team and the project to be more successful and with more quality.

References

Lisa Crispin, Janet Gregory (2009). Agile Testing Practical Guide for Testers and Agile Teams. Publishing by Addison-Wesley