Pre-launch preview · Stepwell Technologies Pvt Ltd (incorporation in progress) · details marked [ ] are being finalised
Home / Insights / Quality engineering
Quality engineering

Flaky tests are a delivery problem, not a QA problem

Google's data and academic research agree on where flakiness comes from. A four-week programme to bring the flake rate down and keep it down.

Key takeaways

  • Google reported that about 16% of its tests showed some flakiness and 84% of pass-to-fail transitions involved a flaky test.
  • Research points to the same root causes: asynchronous waits, concurrency and test order dependency.
  • Measure, quarantine with owners, fix the top causes, then prevent regression: four weeks.
  • Track flake rate as a delivery metric, alongside lead time and change failure rate.

A flaky test passes and fails on the same code. One flaky test is an annoyance. Hundreds of them teach a whole engineering organisation to ignore red builds, and that is when real defects start reaching production.

How common is it?

Google published some of the most cited data. Writing on the Google Testing Blog in 2016, John Micco reported that about 1.5% of all test runs returned a flaky result, that almost 16% of tests showed some level of flakiness, and that 84% of the pass-to-fail transitions observed involved a flaky test.[1]

That last figure is the real cost. If most red builds aren't real failures, rational engineers stop investigating red builds.

Where flakiness comes from

Academic research is consistent on root causes. In an empirical study of flaky tests in open-source projects, Luo and colleagues found the most common categories were asynchronous waits, concurrency and test order dependency.[2] A later survey of the field by Parry and colleagues reviews detection and repair techniques and confirms these as recurring themes.[3] In practice, the usual suspects are:

  • Timing: fixed sleeps, missing waits for asynchronous work, and timeouts tuned on a fast laptop.
  • Shared state: tests that depend on run order, shared databases or leftover data.
  • Environment: unstable test environments, shared third-party sandboxes and network calls.
  • Non-determinism: time zones, random data, unordered collections and floating-point comparisons.

A four-week programme

Week 1: Measure

Record every test result with commit, environment and retry outcome. Calculate a flake rate per test and per suite: runs that both failed and passed on the same commit. Publish the top 20.

Week 2: Quarantine with ownership

Move known-flaky tests out of the merge-blocking path, but never silently. Each quarantined test gets an owner and a fix-by date, and the quarantine list is reviewed weekly. Quarantine without ownership is just deletion with extra steps.

Week 3: Fix the top causes

  • Replace fixed sleeps with condition-based waits. Modern UI frameworks such as Playwright auto-wait for elements to be actionable, which removes a whole class of timing flakes.[4]
  • Make test data hermetic: each test creates and cleans up its own data.
  • Stub third-party services at the boundary; test the real integration in a smaller, separate contract suite.
  • Run tests in random order in CI to expose order dependency early.

Week 4: Prevent regression

Add a flake check for new tests (run them several times before they join the blocking suite), make automatic retries visible in reports rather than hiding them, and set a target such as fewer than 5% of runs affected by flakiness, reviewed monthly.

A test suite people don't trust is worse than no test suite: it costs the same and catches less.

Why leaders should care

Flaky tests slow every merge, waste compute on reruns, and erode the signal that tells you a release is safe. Treat flake rate as a delivery metric next to lead time and change failure rate, not as a QA housekeeping item.

This article is general information, not legal, regulatory or financial advice. Figures are quoted from the public sources listed below as at the publication date; check the source for updates.

Sources

  1. J. Micco, "Flaky Tests at Google and How We Mitigate Them", Google Testing Blog, 27 May 2016. testing.googleblog.com
  2. Q. Luo, F. Hariri, L. Eloussi and D. Marinov, "An Empirical Analysis of Flaky Tests", Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE 2014). doi.org/10.1145/2635868.2635920
  3. O. Parry, G. M. Kapfhammer, M. Hilton and P. McMinn, "A Survey of Flaky Tests", ACM Transactions on Software Engineering and Methodology 31(1), 2021. doi.org/10.1145/3476105
  4. Playwright documentation, "Auto-waiting". playwright.dev