Do You Need More Testers or Better Tests?

Rethinking QA: Navigating the Path to Better Testing Practices

May 13, 2024

Over my decade-long journey of leading software engineering teams, I led or advised multiple transformations from the classic approach of testing (developers develop, testers test) to QA shift-left where attention to quality is an integral part of everyone's job (PMs, engineers), and it starts long before the code is being released (during discovery processes, defining stories or writing the code).

Each of these transformations began in environments with several things in common:

Overall software quality was questionable.
Teams requested hiring more testers.
Software engineers handed off features to QA for testing.
The QA team granted permission to go live.
The "In Testing" task list accumulated.
The SDLC process gets more complicated - there are code freezes and big bang releases. Often, out of 10 changes being released, ticket no. "x" is being reverted to due to its bugs.
There was pressure everywhere: testers had to align automation with new changes and produce detailed reports on what was passing and failing. If a bug was discovered, there were often “war rooms” for engineers where they were expected to deliver "hotfixes" ASAP to avoid further delays.
QA became a separate entity within the company, with its own projects, reports, and dedicated software requiring high-level engineering skills, domain knowledge, and training.

Such a situation is painful for everyone, yet usually, it's so deeply rooted in the culture / ways of working / state of mind that once you try to change it, there is even bigger reluctance.

Engineers are surprised to be expected to test their code.
PMs are reluctant to define clear acceptance criteria.
QA people feel insecure about their jobs.
Product teams (PM + Tech Leader + Engineers) don't know how to approach a mutual discovery process, where both - Product AND Technology should define what is the product/feature they want to build.
Everyone fears delays due to the extra time needed for testing, writing requirements, and defining the desired product features.
The old guard insists that changing ways will only lead to bugs, departures, and product collapse. "Our organization is different," they say, and Continuous Delivery/DevOps Culture/QA Shift-Left won't work here.

This newsletter issue is for leaders facing a similar situation. They seek change but face challenges and resistance that grow their doubts.

Yes - such transformation is painful, costly, and time-consuming, and not everyone will stay during the process. Yet, once completed, it is likely to bring peace and stability to the team, improve product quality, and enhance SDLC performance, all of which impact the company's success.

Here is some food for thought that can help you transform the company's attention to quality and introduce a QA Shift-Left culture. I won't tell you exactly what you should do, but I will bring some ideas and facts about why you should do such a transformation.

For more in-depth stories, I recommend these two pieces I wrote a few years ago:

WEF's Future of Jobs Report

Over the next five years, Software Tester jobs are predicted to decline by at least 15% globally, according to the World Economic Forum's Future of Jobs Report 2023. Meanwhile, Software Engineer jobs are expected to increase by 20-30% (including Data Engineers, DevOps Engineers, Software Engineers, FinTech Engineers, and others).

The report doesn't detail the reasons behind these trends, but some industry shifts offer insight. For example, according to the DORA research, the highest-performing engineering teams achieve a change lead time (from commit to deploy) of less than one day, have on-demand deployment frequency, and a failure rate of <5%. Testing software manually at such a rapid pace is virtually impossible. Even if only new features are tested manually, coordination between engineers and testers would need to be perfect.

Microsoft Retired Software Development Engineer in Test a Decade Ago

In a recent article, Gergely Orosz wrote about How Microsoft Does Quality Assurance (QA). Microsoft initially pioneered the role of Software Development Engineer in Test (SDET), focusing on automated testing and building testing systems to foster a strong testing culture.

However, challenges arose that many of us know:

Who should write unit tests, engineers or testers?
"Us vs. them" dynamics, with differing goals, schedules, QA gatekeeping, and "ticket ping-pong."
Complex testing systems that still required senior engineering help.
Inverted testing pyramid with the majority of heavy E2E tests.

These challenges and others led Microsoft to retire the SDET role in 2014, favoring a unified Software Engineer role combining development and testing responsibilities. This change streamlined workflows, improved code quality, and reduced the divisive dynamics between development and testing.

The evolution of tests in Microsoft, screenshot from Pragmatic Engineer’s newsletter.

This merger led to better testing strategies, focusing more on the granular unit and integration tests than time-consuming end-to-end tests. The change resulted in improved quality, agility, and engineer satisfaction. This shift was also well-suited for Microsoft's increasing focus on SaaS models, helping teams ship products more frequently without compromising quality.

DevOps Culture

In previous articles, I explored various aspects of DevOps culture. Inspired by Lean Manufacturing and the Toyota Production System, DevOps culture streamlines workflows, emphasizes automation, and promotes frequent, small-batch releases.

In the DevOps Culture Checklist, the focus is on:

Quick, routine, and stress-free deployment processes.
Fast, comprehensive tests providing feedback within minutes.
Production telemetry detecting problems early.
Automated tests integrated into daily workflows.
Everyone owning quality and writing tests for their work.

Manual testing and tests created outside of the regular development cycle cause several types of Software Engineering Waste:

Handoffs between engineers and testers lead to knowledge loss and missing context.
Context switching related to handoffs and the time engineers wait for QA feedback (sometimes days, weeks) lead to cognitive overload, where people need to switch their attention from features they develop to features they must fix after testing.
Waiting: If engineers receive feedback in five minutes, they will likely remain focused on the current context. If they must wait days, their attention will likely drift.

According to the DevOps Handbook, inspired by Toyota's TPS, processes should be built so defects are swarmed and fixed immediately. Like the "Andon cord" principle, where workers stop the production line when something goes wrong, our SDLC should enable fixing issues when they are created (development) rather than after they are pushed to customers.

If you learn about issues from your customers, it's bad. It can take weeks to discover them.
It's slightly better if you learn about issues from manual tests. However, this feedback can take days to receive.
If you catch issues in automation testing, it's great. With an optimized process, it can take minutes to learn about issues.

Image from https://www.stickyminds.com/article/shift-left-approach-software-testing

Why? According to Applied Software Measurement, the cost of fixing bugs grows exponentially the longer they remain undetected.

"It’s impossible for a developer to learn anything when someone yells at them for something they broke six months ago—that’s why we need to provide feedback to everyone as quickly as possible, in minutes, not months."
— DevOps Handbook

Developer Experience

Let's change the angle slightly. In transforming your company's approach to testing, some might say that requiring engineers to write tests will decrease satisfaction and drive them away. No one wants to test things, everyone wants to create, no?

In recent years, DevEx practices have gained traction, encouraging organizations to improve Developer Experience and productivity. Abi Noda’s research DevEx: What Actually Drives Productivity identifies three dimensions of developer experience:

Flow State
Feedback Loops
Cognitive Load

These dimensions encompass various types of friction we should reduce as engineering leaders.

Screenshot from Abi Noda’s research DevEx: What Actually Drives Productivity

The SDLC process directly impacts all these dimensions. Unnecessary handoffs, context switching, and waiting days for QA feedback all interfere with Flow State, Feedback Loops, and Cognitive Load. Thus, writing tests directly may seem unappealing short-term, but in the long run, it can increase satisfaction and productivity. Staying focused allows engineers to maintain a flow state and reduce cognitive load, while quicker feedback from automated testing or production telemetry ensures they can deliver bug-free software efficiently.

What About Testers and QA Engineers?

Using Microsoft's story as an example, the role of testers was introduced in a different era, when software release cycles spanned months or years and products were distributed via boxed CD-ROMs.

We waited at least a year to get a new version of Windows or Photoshop. Today, Adobe Lightroom, which I use for my photography hobby, is released multiple times a month, if not daily, often without my notice.

This shift also demands changes to the development process. Just as Toyota streamlined production in 1945 by placing machines close together and syncing speeds to maintain a consistent pace, today's development workflows integrate quality and security checks.

So, what happens to testers who may become redundant? Such transformations don't occur overnight and often take months or years to phase out the QA team, providing ample time to rethink the role's future.

Manual testers often transition to QA engineers, automating work that takes hours into scripts executed in minutes. Their role may then evolve into that of an evangelist or agent of change, creating solutions (templates, infrastructure) that all engineers can use. They ensure every developer knows how to run tests and can do so in minutes.

Over time, it may become an infrastructure engineering work. E.g., making CI/CD pipeline optimizations to cut testing execution time by tests sharding and parallelizing (a few years ago, one of my team reduced execution time from a few hours to < 30 minutes by moving testing infrastructure to the cloud solutions).

Wait... does this mean a QA Engineer becomes a Software Engineer? In short, yes. Many QA engineers already do engineering work: writing scripts, setting up testing infrastructure, building automated pipelines—it's also code development. Yet many organizations keep dedicated QA Engineer roles, often paying them 50% - 70% of the engineer's salary and expecting less from them than from regular engineers (you are good at writing automations but not so good at writing production code).

And what about "special tester's eye"? I don't believe such a thing exists. It's nothing different than attention to detail and a sense of ownership. No one knows your code better than you if you are a software engineer. There can gaps in requirements and acceptance criteria. There could be a misunderstanding between Product and Technology regarding what the customers value we build. But none of those justify someone else telling you for an extended period of time how your product should behave and why it doesn't right now.

We have pair programming, peer reviews, team demo sessions, feature flags, and internal testing. You can ask your PM, UX designer, or the entire team to play with the app you build. Do you think having an additional line of testers is necessary here?

"In complex systems, adding more inspection steps and approval processes actually increases the likelihood of future failures. The effectiveness of approval processes decreases as we push decision-making further away from where the work is performed."
— DevOps Handbook

Final Thoughts

Does this mean the Software Tester role will disappear soon? Unlikely. Even though Microsoft retired the role a decade ago and the term DevOps culture existed even before that, many organizations still follow traditional quality control practices.

However, research like WEF's Future of Jobs Report and DORA's studies indicate a clear trend: testers are becoming a niche or transforming into software engineers. DORA research also highlights the stark difference between organizations that have streamlined processes and those still following traditional methods. The latter has a clear impact on company performance.

This transformation may be difficult and fraught with challenges and resistance, causing some teammates short-term discomfort. But in the long run, it's worth it. The long-term benefits will yield better results for your team, your product, and your company.

What do you think?