← pytest: reliable Python tests Progress & review ↗

Typorithm · medium

Skips and strict expected failures

pythonpytesttestingindependent-practice

Saved on this device

Full lesson · 4 learning steps Open any exercise for focused practice

Step 1 of 4 · intro card

Skips and strict expected failures

Concept and contract

Report limitations honestly and surface an unexpectedly fixed defect.

A skip says a test is not applicable or cannot run under a documented condition. An xfail says a known defect is expected to make the test fail. These communicate different evidence: skipped assertions did not validate the feature, while an expected failure records a known problem.

Use an explicit reason and a reviewable condition. A permanent unconditional skip with no owner can become forgotten missing coverage. Likewise, xfail should not be a blanket response to a mysterious flaky test that nobody has investigated.

Mechanism and worked example

The worked example includes a deliberately broken legacy decoder with a strict expected-failure test.

The worked example includes a deliberately broken legacy decoder with a strict expected-failure test. The test still runs, fails and is reported as xfailed. A separate current decoder is tested normally. strict=True makes an unexpected pass fail the session so the known-defect marker must be reviewed.

The raises argument narrows the expected failure type to AssertionError in this example. It prevents an unrelated error type from satisfying the marker. It does not prove that the precise assertion failed for the intended underlying reason; keep the test body small and specific.

Complete worked example

python

import pytest


def legacy_decode(value):
    return value


def current_decode(value):
    return value.upper()


@pytest.mark.xfail(strict=True, raises=AssertionError, reason="Owned teaching defect: legacy decoder omits uppercase")
def test_known_legacy_defect():
    assert legacy_decode("sql") == "SQL"


def test_current_decoder():
    assert current_decode("sql") == "SQL"

Reasoning: The worked example includes a deliberately broken legacy decoder with a strict expected-failure test. The test still runs, fails and is reported as xfailed. A separate current decoder is tested normally. strict=True makes an unexpected pass fail the session so the known-defect marker must be reviewed. The raises argument narrows the expected failure type to AssertionError in this example. It prevents an unrelated error type from satisfying the marker. It does not prove that the precise assertion failed for the intended underlying reason; keep the test body small and specific.

Open in local Python playground ↗

Focused test decision: context is in the complete example

python

@pytest.mark.xfail(strict=True, raises=AssertionError, reason="Known defect")
def test_known_legacy_defect():
    assert legacy_decode("sql") == "SQL"

Focused test decision: Type this short test decision after understanding the complete runnable module. Desktop typing records practice; Run tests in the local playground executes real pytest.

Counterexamples and common mistakes

The controlled counterexample fixes the legacy decoder but leaves its strict xfail marker.

The controlled counterexample fixes the legacy decoder but leaves its strict xfail marker. pytest reports XPASS(strict) and returns a failing result. This is intentional: the suite asks the maintainer to remove or update the obsolete expected-failure declaration.

Without strict behavior, an unexpected pass can remain merely informational. Do not generalize that setting as always wrong; teams may choose a policy. For a small teaching suite, strict xfail makes the review signal explicit and keeps known limitations visible.

Controlled bug: the tests must report failure

python

import pytest


def legacy_decode(value):
    return value.upper()


def current_decode(value):
    return value.upper()


@pytest.mark.xfail(strict=True, raises=AssertionError, reason="Owned teaching defect: legacy decoder omits uppercase")
def test_known_legacy_defect():
    assert legacy_decode("sql") == "SQL"


def test_current_decoder():
    assert current_decode("sql") == "SQL"

Reasoning: The controlled counterexample fixes the legacy decoder but leaves its strict xfail marker. pytest reports XPASS(strict) and returns a failing result. This is intentional: the suite asks the maintainer to remove or update the obsolete expected-failure declaration. Without strict behavior, an unexpected pass can remain merely informational. Do not generalize that setting as always wrong; teams may choose a policy. For a small teaching suite, strict xfail makes the review signal explicit and keeps known limitations visible.

Open in local Python playground ↗

Independent practice

Add a normal empty-input case for the current decoder.

Add a normal empty-input case for the current decoder. Do not mark the new test xfail: the documented current API already supports empty text. Keep the legacy expected-failure example separate from supported behavior.

Try the controlled fixed-legacy version and inspect the report before returning to the baseline. Then explain why turning a failing supported case into a skip would conceal a regression rather than repair it.

Step 1: Add a normal empty-input test for the supported current decoder. Step 2: Keep the existing passing cases. Add the smallest new assertion first; predict the failure before running pytest.

Practice starter: solve before reading the solution

python

import pytest


def legacy_decode(value):
    return value


def current_decode(value):
    return value.upper()


@pytest.mark.xfail(strict=True, raises=AssertionError, reason="Owned teaching defect: legacy decoder omits uppercase")
def test_known_legacy_defect():
    assert legacy_decode("sql") == "SQL"


def test_current_decoder():
    assert current_decode("sql") == "SQL"

# Task: Add a normal empty-input test for the supported current decoder.

Reasoning: Add a normal empty-input test for the supported current decoder.

Open in local Python playground ↗

Hint 1: Use current_decode, not the deliberately broken legacy function.
Hint 2: The new supported behavior should pass without an xfail marker.

Solution, tests and trade-offs

The solution adds the current decoder’s empty boundary.

The solution adds the current decoder’s empty boundary. The practice probe returns a fallback only for empty text. The existing current happy path and known legacy xfail stay as before; the new normal test fails.

A green session containing xfails and skips is not equivalent to every feature passing. Report the counts and reasons. In the playground the terminal is the authoritative summary; phase reports show skip outcomes, and the reason distinguishes an expected defect from a conditionally inapplicable test.

Practice solution and acceptance checks

python

import pytest


def legacy_decode(value):
    return value


def current_decode(value):
    return value.upper()


@pytest.mark.xfail(strict=True, raises=AssertionError, reason="Owned teaching defect: legacy decoder omits uppercase")
def test_known_legacy_defect():
    assert legacy_decode("sql") == "SQL"


def test_current_decoder():
    assert current_decode("sql") == "SQL"

def test_current_decoder_preserves_empty():
    assert current_decode("") == ""

Reasoning: The solution adds the current decoder’s empty boundary. The practice probe returns a fallback only for empty text. The existing current happy path and known legacy xfail stay as before; the new normal test fails. A green session containing xfails and skips is not equivalent to every feature passing. Report the counts and reasons. In the playground the terminal is the authoritative summary; phase reports show skip outcomes, and the reason distinguishes an expected defect from a conditionally inapplicable test.

Open in local Python playground ↗

Interview reasoning and next step

Explain skip versus xfail, strict unexpected-pass behavior and the limits of specifying raises.

Explain skip versus xfail, strict unexpected-pass behavior and the limits of specifying raises. Describe a process for removing a marker after the defect is fixed, including retaining a normal regression test.

Next test a real owned SQLite boundary. Environmental limitations should be stated honestly rather than hidden with a marker that makes a browser-only demonstration claim support for a production database. Keep marker reasons specific enough that a maintainer can decide when they have become obsolete, rather than using a generic “broken” label.

Optional stretch — after the core task works: Introduce one deliberate implementation mistake and explain which assertion catches it. A green run alone is not evidence that every possible bug is covered.

Computer lab: https://applydjinn.com/typorithm/pytest-lab-20261106.zip. Run verify.py in an isolated environment. Worked examples and solutions pass, controlled counterexamples fail, and the new practice assertions reject a mutation missed by the baseline. Run --practice pyq-xfail after changing the supplied exercise. Python executes on your computer or in the browser, without submitting code to a server. Native question progress is a separate learning record.

Report limitations honestly and surface an unexpectedly fixed defect.