Flaky tests are the quiet killer of CI confidence. A test passes locally, fails on main, passes again on rerun, and no one knows why. Teams start ignoring red builds, merge broken code, and eventually stop trusting the suite.

This post is about finding and fixing flaky end-to-end tests, not about writing more tests. It covers isolation, timing, deterministic data, and a quarantine workflow that keeps the suite honest.

Why tests flake

End-to-end tests touch the most moving parts: browser, network, database, third-party APIs, and system clock. Common causes:

  • Shared state between tests: one test creates a user, another deletes it.
  • Fixed sleeps instead of waiting for a condition.
  • Race conditions between UI updates and assertions.
  • Tests that depend on running in a specific order.
  • External services that are slow or return random data.
  • Dates and timestamps that change between runs.

Reproduce locally before you fix anything

Do not guess. Reproduce the failure on your machine first.

Run the failing test repeatedly:

bash
pytest tests/e2e/test_billing.py::test_invoice_payment --count=50

If your runner supports it, shuffle the suite to catch ordering bugs:

bash
pytest tests/e2e --randomly-seed=42

When it fails, capture the browser trace, screenshot, and console logs. In Playwright you can enable tracing per test:

python
from playwright.sync_api import sync_playwright

def test_flaky_scenario():
    with sync_playwright() as p:
        browser = p.chromium.launch()
        context = browser.new_context()
        context.tracing.start(screenshots=True, snapshots=True)
        page = context.new_page()
        page.goto('https://example.invalid/dashboard')
        page.click('button.submit')
        context.tracing.stop(path='trace.zip')
        browser.close()

If you cannot reproduce locally, run the same test in a CI-like container with the same browser version and database image. Differences in Chrome versions or Postgres time zones are a common source of works on my machine.

Isolate state between tests

Shared state is the most common cause of flakiness. Every test should start from a clean slate.

Say a team has a test that creates a project and then asserts it appears in the list. If the previous test deleted all projects, fine. If the previous test created a project with the same slug, the new test may fail with a duplicate key or assert against the wrong row.

Use unique data per test:

python
import uuid

def test_project_creation(page, db):
    slug = f'proj-{uuid.uuid4().hex[:8]}'
    page.fill('input.slug', slug)
    page.click('button.create')
    expect(page.locator(f'text={slug}')).to_be_visible()
    db.execute('DELETE FROM projects WHERE slug = %s', (slug,))

Better still, wrap each test in a database transaction and roll it back:

python
@pytest.fixture
def clean_db(db_connection):
    db_connection.begin()
    yield
    db_connection.rollback()

For browser tests, use a fresh context per test, not a shared page object. Do not store cookies or local storage across tests unless you are explicitly testing persistence.

Remove timing dependence

Replace fixed sleeps with assertions that wait for a condition.

Bad:

python
page.click('button.save')
time.sleep(2)
assert page.is_visible('.toast.success')

Good:

python
page.click('button.save')
expect(page.locator('.toast.success')).to_be_visible()

Playwright's expect retries until the condition is true or a timeout is reached. If you are using raw Selenium or Cypress, use explicit waits instead of sleep.

Beware of assertions that check exact text including timestamps:

python
# Brittle
assert page.inner_text('.last-synced') == 'Synced at 14:32:05'

# Robust
expect(page.locator('.last-synced')).to_contain_text('Synced at')

If your application depends on the clock, freeze time in tests. For Python, freezegun is a common choice:

python
from freezegun import freeze_time

@freeze_time('2024-01-15 12:00:00')
def test_subscription_expiry(client):
    ...

Make test data deterministic

Tests that read the latest order or the first user fail when another test inserts data first. Query by a known identifier instead.

python
# Brittle
order = Order.query.first()
assert order.status == 'paid'

# Robust
order = Order.query.filter_by(reference='ORD-TEST-001').one()
assert order.status == 'paid'

Avoid real third-party APIs in E2E tests unless you are specifically testing that integration. Use recorded responses or a local mock server. If you must call a real API, sandbox the account and seed it with known data before every run.

Test ordering and parallelism

CI often runs tests in parallel across workers. If two tests write to the same file, table, or S3 bucket, they will collide.

Run your suite locally in random order to surface ordering bugs:

bash
pytest tests/e2e --randomly

If a test only fails when run after another, you have found shared state. Fix it by giving each test its own resources or by serializing the conflicting tests with a marker:

python
@pytest.mark.serial
def test_billing_webhook(client):
    ...

In CI, be careful with sharding. Splitting tests by file is safer than splitting by test ID when tests share setup fixtures.

Retries are a safety net, not a fix

Most test runners let you retry a failed test once before marking it red. Use this to reduce noise, but never as a substitute for fixing the root cause.

Example GitHub Actions step with pytest retry:

yaml
- name: Run E2E tests
  run: |
    pip install pytest-rerunfailures
    pytest tests/e2e --reruns 1 --reruns-delay 2

If a test passes on retry, create a ticket immediately. Tag it flaky and assign an owner. If it is not fixed within a sprint, quarantine it:

python
@pytest.mark.flaky(reason='GH-1234: intermittent DNS timeout', until='2025-02-01')
def test_external_redirect(client):
    ...

Quarantined tests should run in a separate job so they do not block merges, but they still run so you know when they start passing consistently, or when they get worse.

A quarantine workflow

  1. Detect: test fails in CI, passes on rerun.
  2. Reproduce: run locally with --count and random order.
  3. Isolate: find shared state, timing, or external dependency.
  4. Fix: add unique data, explicit waits, or mocks.
  5. Verify: run the full suite locally and in CI five times.
  6. Remove quarantine: delete the retry marker or flaky tag.

If you cannot fix it quickly, disable the test rather than leaving it flaky. A missing test is visible; a flaky test is deceptive.

Checklist

  • Each test creates and cleans up its own data.
  • No time.sleep, cy.wait(5000), or fixed pauses.
  • Assertions wait for UI state, not wall-clock time.
  • Tests pass when run in random order.
  • External APIs are mocked or sandboxed.
  • Flaky tests have a ticket and an owner.
  • Retries are enabled in CI but tracked as technical debt.

Takeaway

Flakiness is usually an architecture problem in the test suite, not a browser problem. Fix isolation and timing first, use retries only as a temporary safety net, and quarantine anything you cannot fix immediately so the rest of the suite stays trustworthy.

IFLEON can help teams harden their test automation and CI pipelines.