A few hundred sols ago we explored using some type of automated testing to speed up our testing efforts which were largely manual and semi-automated. Turn around times for complete regression testing ranged from days to weeks which was a major impediment to quick release cycles. It was also an exhausting task for our QAs while leaving the DEVs guessing the impact of all those code changes that went into the release. Yes, the unit tests worked but that was just a small little warm-fuzzy and did not prove nothing bigger was broken.
Usually when we talk about testing the following stages come to mind:
Unit tests: The scope of these tests is very localized to the feature or fix that is coded. You may run a larger set of unit tests at the product level to ensure nothing within the product broke. It provides relatively quick and immediate feed back so you instantly, or within a few minutes, know if something is broken and can fix it. External services that your product interacts with are typically mocked in these tests.
Regression tests: Typically involves testing a larger set of scenarios this time with real instances of external services where applicable and others mocked. This may happen after a code build via a CI/CD tool and deployed to an automated test environment
User Acceptance Testing: This involves the customer or an external user of the product running some tests to ensure their use-cases work as expected. Maybe some performance testing is involved in this phase as well but rarely?
The key here is that as testing moves farther and farther away from the developer’s domain it becomes increasingly expensive to fix it in terms of time taken. When failures happen close to the development stage it is easy to notice and fix them. Once it is “delivered over the fence” to professional testers the time factor to detect and fix errors increases. This could substantially affect the quality of the product because the developers have “moved on” to the next set of features/releases and lose the context of the previous release which may no longer be fresh in their mind so it can be effectively fixed.
The important thing here is the speed of feedback. It goes without saying that first of all, automated testing is the only way to accomplish this and second of all it needs to happen close to the development cycle, preferably in the “gap” between when it is trying to move from the developer’s domain to the QAs. That way the developers do not need to wait for an inordinate amount of time for full feedback on their code.
This automated testing could also be a common reference point between the developers and the QA teams so that its outcome would trigger the same confidence levels on the build for both parties. This was important because otherwise the developers would call it a day after all their unit-tests passed and now the “ball was in the QA’s courts” so to speak.
The implication then, when a shared test suite becomes important to both parties, is that they share a common language in which to express the tests. It not only makes it clear what is being tested but also facilitates the re-creation of a test in any environment without ambiguity. This is where we stumbled upon the world of BDD and test expression languages like Gherkin and its implementation in various forms (Java, Python). This was a game changer in terms of how we approached the whole testing process.
For us testing both at the lowest level (unit tests, “are we building the system right?”) vs business level testing (QA testing, “are we building the right system?”) was important. We already had automation at the unit test level but not at the QA level who typically worked more on the business level. Moving to BDD facilitated automating business level testing as well which saved a huge amount of time at what was typically a large time-consuming activity. Testing times came down from several days to weeks to under 24 hours.
BDD can span multiple domains: developers, testers, business stake holders. It does take organizational buy-in for it to permeate into the various layers and to result in a “group think”mindset where everyone is thinking “testing first”. This may not be an easy sell in an organization due to the heterogenity of processes and nature of product or service provided. In most cases it may be prudent to introduce it in little steps affecting the smallest number of persons/teams to see how effective it can or cannot be!
The scope and techniques of BDD is too big to be discussed in one blog post. Here is a place to get started for more insight! Hopefully I can return for a second part on this topic and discuss some of the best practices that I think would be helpful for someone interested in exploring this in their own workspace!
sidespin.blog 