What are flaky tests, and how do you fix them?

Your suite is red, but nobody bothers checking anymore. That moment the team decides to ignore it, that's the real problem.

Everyone knows the scenario. The build fails. Someone looks, shrugs, and hits "rerun". Second time it passes. Problem "solved".

That moment when the team decides that red no longer means something is broken, that's the real problem. Not the failing test. Getting used to it.

What is a flaky test?

A flaky test is a test that sometimes passes and sometimes fails, without the code changing. Same input, same environment, different result. It doesn't consistently detect the same behaviour. And that makes it more dangerous than a test that fails every time.

A broken test you can fix. An unreliable test undermines something more fundamental: trust in your entire suite. Because if this test sometimes lies, which others do too?

The real danger

Flaky tests create a false sense of security. Your suite is green, so everything's fine. Except part of that green is luck. And the reverse is worse: teams learn to ignore red. "Ah, that one's just flaky." Until the day red really was red.

We see with clients how this creeps in. First it's one test. Then three. Then there's a shared understanding on the team that "the suite is always a bit red". At that point your safety net isn't a safety net anymore, it's decoration.

The five causes

Timing

The classic. The test clicks a button before the page has finished loading. Fine on a fast machine, not on a slow CI runner. Fixed waits (sleep(2)) are a patch that eventually comes loose.

External dependencies

Your test depends on systems outside its control: an external API, a service from another application, or a shared environment. It's briefly slow, briefly down, or returns something slightly different. Your test fails, but your code is fine.

Shared test data

Two tests use the same record. One changes it, the other reads it. Run them in parallel and the outcome is a gamble. Run them in a different order, same problem.

Unreliable environments

The test environment isn't quite the same as production. Something from yesterday is still running. The cache isn't empty. One runner has more memory than another.

Non-deterministic logic

The test depends on the current time, random data, or the order in which tests run. As a result, the exact same test can give different results.

How to fix them

• Isolate your tests. No shared state between test cases. Each test creates its own data and cleans up after itself.
• Mock external dependencies. Mock them wherever reliability matters more than the integration itself. Your test shouldn't fail randomly because a third party is briefly unavailable.
• Avoid fixed waits. Wait on a condition ("element is visible"), not on a number of seconds.
• Track which tests are flaky. Measure it. A test that failed incorrectly three times this month is a bug.
• Make your tests deterministic. Use fixed test data, control time-dependent behaviour with a mock or fake clock, and avoid random values or dependencies on execution order.

The difference between the two comes down to a few lines of code:

// flaky: guesses how long loading takes
await page.click('#submit');
await page.waitForTimeout(2000);
await expect(page.locator('#confirmation')).toBeVisible();

// stable: waits on the condition itself, not a guess
await page.click('#submit');
await expect(page.locator('#confirmation')).toBeVisible({ timeout: 10000 });

On a fast machine you won't see the difference. On a busy CI runner, during a hundred parallel builds, you will — and that's exactly when you want your test waiting on what's actually happening, not on an assumption.

The attitude that makes the difference

The most important thing isn't technique, it's a mindset shift: treat a flaky test as a bug. Not as background noise, not as "it just does that sometimes". Put it on the backlog, give it a priority, and fix it.

Teams that do this end up with a suite where red really means red. And in the end, that's the only thing that matters.