13. Flaky tests, traces and reproducible failures

qatestingplaywrightindependent-practice

Saved on this device

Full lesson · 4 learning steps Open any exercise for focused practice

Step 1 of 4 · intro card

13. Flaky tests, traces and reproducible failures

Concept and contract

Investigate inconsistent outcomes instead of converting them into silent passes.

A flaky test produces inconsistent outcomes under conditions intended to be equivalent. Possible causes include shared data, timing assumptions, environmental differences and genuine application races. A retry can reveal that inconsistency, but it does not establish which cause is responsible.

Preserve the failing input, relevant environment and bounded diagnostics. A trace can show actions, network observations and DOM snapshots. Real traces may contain sensitive data, so retention and access are part of the test design rather than an afterthought.

Mechanism and worked example

The worked test waits for the complete fixture and attaches a small nonsecret observation containing only role ids.

The worked test waits for the complete fixture and attaches a small nonsecret observation containing only role ids. That evidence helps compare runs without dumping account cookies or full payloads. It complements, rather than replaces, the assertion.

For diagnosis, rerun one case with the same fixture, then vary one suspected factor. Compare serial and bounded-parallel execution when shared state is suspected. Do not change the oracle merely because a failure is inconvenient.

Complete worked example

typescript

import { test, expect } from "../support/test.js";
test("records useful fixture evidence", async ({page}, testInfo) => {
  await page.goto("/");
  const rows = page.getByRole("list", {name:"Matching roles"}).getByRole("listitem");
  await expect(rows).toHaveCount(4);
  const ids = await rows.evaluateAll(items => items.map(item => item.getAttribute("data-id")));
  await testInfo.attach("fixture-ids", {body:JSON.stringify({ids}),contentType:"application/json"});
  expect(ids).toEqual(["r1","r2","r3","r4"]);
});

Reasoning: The attachment records a small synthetic observation without swallowing the assertion.

Counterexamples and common mistakes

The weak pattern catches an assertion error and logs a warning while allowing success.

The weak pattern catches an assertion error and logs a warning while allowing success. It destroys the runner’s failure signal. A retry count of many can have a similar reporting effect when the final green status hides repeated earlier failures.

Do not add long sleeps as the first response to every flaky result. Identify the missing readiness condition, shared resource or actual race. A useful fix should include a regression that would fail for the diagnosed cause.

Counterexample: what this check misses

typescript

// Weak pattern: catch an assertion error, log a warning, and let the test pass.
// The failure signal must reach the runner.

Reasoning: Logging and continuing can convert a real failed contract into a false green result.

Independent practice: small task

Write a test that records expected and actual fixture ids as a JSON attachment and then asserts their equality.

Write a test that records expected and actual fixture ids as a JSON attachment and then asserts their equality. Keep the attachment limited to synthetic ids. Begin with the assertion before adding diagnostic context.

Optional extension: run the controlled OR-filter mutant and inspect its failure artifact. Keep that deliberately red run separate from successful verification so it cannot be mistaken for an unresolved production defect.

Step 1: Keep the independent id assertion. Step 2: Attach only the fixture context needed to investigate a mismatch.

Practice starter: solve before reading the solution

typescript

import { test, expect } from "../support/test.js";
test("diagnostic evidence retains the assertion", async ({request}, testInfo) => {
  // TODO: attach only synthetic ids, then assert the independent expected list.
});

Reasoning: Write the independent assertion first, then add only useful nonsecret context.

Hint 1: Do not catch and hide assertion failures.
Hint 2: A trace can contain sensitive data.

Solution, tests and trade-offs

The solution attaches a compact fixture observation through testInfo and leaves the assertion failure visible.

The solution attaches a compact fixture observation through testInfo and leaves the assertion failure visible. The lab verifier also performs a controlled failing browser run to demonstrate that the combined-filter test can reject a wrong implementation.

An attachment’s existence does not prove a cause. Read the evidence and connect it to a specific violated contract. The successful suite and expected failing mutation are reported as different outcomes.

Practice solution and acceptance checks

typescript

import { test, expect } from "../support/test.js";
test("diagnostic evidence retains the assertion", async ({request}, testInfo) => {
  const response = await request.get("/api/roles?country=UA&q=qa");
  const data = await response.json();
  const ids = data.items.map((role:{id:string}) => role.id);
  await testInfo.attach("query-evidence", {body:JSON.stringify({expected:["r1"],actual:ids}),contentType:"application/json"});
  expect(ids).toEqual(["r1"]);
});

Reasoning: Expected and actual ids are retained while a mismatch still fails the test.

Interview reasoning and next step

In interviews, describe a minimal reproduction and one hypothesis you would test.

In interviews, describe a minimal reproduction and one hypothesis you would test. Explain how retries, traces and fixture isolation contribute different evidence. State what information should not enter a public artifact.

Next, run the same bounded checks in CI without adding secrets, uncontrolled concurrency or an ambiguous green status.

Optional stretch — after the core task works: Inspect the controlled failing filter run separately from the green suite.

Computer practice: download the companion lab at https://applydjinn.com/typorithm/qa-lab-20261105.zip. Read its README, install the pinned tools and run the accepted examples before attempting a starter. The mobile reader preserves your reading and self-check progress; the local lab runs browser tests separately.

Investigate inconsistent outcomes instead of converting them into silent passes.