HTTP failures versus transport failures

pythonapi-clientsautomationindependent-practice

Saved on this device

Full lesson · 4 learning steps Open any exercise for focused practice

Step 1 of 4 · intro card

HTTP failures versus transport failures

Concept and contract

Preserve status evidence by catching HTTPError before its URLError superclass.

An HTTP error response and a failed transport are different observations. A server can successfully send a 404 or 429 response, while a connection failure may produce no HTTP status at all.

urllib models HTTPError as a subclass of URLError, so exception ordering matters. This lesson produces a small explicit status record and deliberately avoids guessing an HTTP status for a failure that never received one.

Mechanism and worked example

The try block closes a successful response through its context manager.

The try block closes a successful response through its context manager. The HTTPError branch records its actual code and closes the error response before returning.

Only the later URLError branch represents the narrower remaining transport category. Automatic redirects remain disabled, so a redirect is visible as an HTTP outcome under this lesson’s policy. Unexpected exceptions are not converted into a misleading successful or empty result.

Complete worked example

python

from urllib.request import build_opener, HTTPRedirectHandler, ProxyHandler

class NoRedirect(HTTPRedirectHandler):
    def redirect_request(self, request, file, code, message, headers, new_url):
        return None

from urllib.error import HTTPError, URLError

def response_status(url):
    try:
        with build_opener(ProxyHandler({}), NoRedirect()).open(url, timeout=2) as response:
            return {"kind": "ok", "status": response.status}
    except HTTPError as error:
        status = error.code
        error.close()
        return {"kind": "http", "status": status}
    except URLError:
        return {"kind": "transport", "status": None}

Reasoning: Preserve status evidence by catching HTTPError before its URLError superclass. Compare the stated values and failure boundary before changing the implementation.

Counterexamples and common mistakes

The counterexample catches URLError first.

The counterexample catches URLError first. Because HTTPError inherits from it, a received 429 is then mislabeled as a transport failure and its useful status evidence disappears.

A broad except Exception returning an empty list is worse: it can turn an import failure into an apparent empty dataset. Do not translate lack of response into a fabricated 500 status; that number would claim evidence from a server response you did not observe.

Counterexample: a runnable policy defect

python

from urllib.request import build_opener, HTTPRedirectHandler, ProxyHandler

class NoRedirect(HTTPRedirectHandler):
    def redirect_request(self, request, file, code, message, headers, new_url):
        return None

from urllib.error import HTTPError, URLError

def response_status(url):
    try:
        with build_opener(ProxyHandler({}), NoRedirect()).open(url, timeout=2) as response:
            return {"kind": "ok", "status": response.status}
    except URLError:
        return {"kind": "transport", "status": None}
    except HTTPError as error:
        status = error.code
        error.close()
        return {"kind": "http", "status": status}
    except URLError:
        return {"kind": "transport", "status": None}

Reasoning: This runnable counterexample violates the declared contract; the companion lab requires it to fail.

Independent practice: API client task

Add a retryable boolean to each result.

Add a retryable boolean to each result. Mark only received statuses 429, 502, 503 and 504 as retryable; leave success and the transport category false in this smaller policy.

Begin with 429 and 404, then add success and transport cases. Whether a transport failure should be retried requires a separate idempotency and budget decision, so the task does not guess that decision for every possible caller.

Step 1: Add the declared retryable boolean to each category. Step 2: Keep the exact received HTTP status and closure behavior.

Practice starter: solve before reading the solution

python

from urllib.request import build_opener, HTTPRedirectHandler, ProxyHandler

class NoRedirect(HTTPRedirectHandler):
    def redirect_request(self, request, file, code, message, headers, new_url):
        return None

from urllib.error import HTTPError, URLError

def response_status(url):
    # Remove this placeholder, then adapt the retained baseline:
    # Add the declared retryable boolean to each category.
    raise NotImplementedError
    try:
        with build_opener(ProxyHandler({}), NoRedirect()).open(url, timeout=2) as response:
            return {"kind": "ok", "status": response.status}
    except HTTPError as error:
        status = error.code
        error.close()
        return {"kind": "http", "status": status}
    except URLError:
        return {"kind": "transport", "status": None}

Reasoning: Add the declared retryable boolean to each category. Keep the exact received HTTP status and closure behavior.

Hint 1: Use membership in the explicit four-status set.
Hint 2: Do not invent a status or automatically retry a transport failure.

Solution, tests and trade-offs

The fixture server returns controlled success, missing-resource and overload responses.

The fixture server returns controlled success, missing-resource and overload responses. A deliberately unsupported URL scheme exercises URLError without contacting an external host.

Cases compare exact category and status values, preventing an implementation from presenting every failure as the same generic record. The practice checks the declared retryable set, including a false result for 404 and 500. A policy label does not itself perform a retry or wait.

Practice solution and acceptance checks

python

from urllib.request import build_opener, HTTPRedirectHandler, ProxyHandler

class NoRedirect(HTTPRedirectHandler):
    def redirect_request(self, request, file, code, message, headers, new_url):
        return None

from urllib.error import HTTPError, URLError

def response_status(url):
    try:
        with build_opener(ProxyHandler({}), NoRedirect()).open(url, timeout=2) as response:
            return {"kind": "ok", "status": response.status, "retryable": False}
    except HTTPError as error:
        status = error.code
        error.close()
        return {"kind": "http", "status": status, "retryable": status in {429, 502, 503, 504}}
    except URLError:
        return {"kind": "transport", "status": None, "retryable": False}

Reasoning: Use membership in the explicit four-status set. Do not invent a status or automatically retry a transport failure.

Interview reasoning and next step

In an interview, trace the exception hierarchy and explain where response evidence is retained.

In an interview, trace the exception hierarchy and explain where response evidence is retained. Then distinguish an error classification from an operational retry policy.

A real client may also expose a timeout category, certificate-validation failure and a user cancellation, each with different handling. The next lesson uses a deliberately limited set of statuses to calculate a bounded delay without sleeping or issuing repeated production requests.

Optional stretch — after the core task works: Design a separate timeout category and explain which evidence distinguishes it.

Computer practice: https://applydjinn.com/typorithm/python-automation-lab-20261105.zip. Run verify.py using your selected Python interpreter, then adapt the retained exercise baseline and check it with --practice followed by the lesson id. The standard-library lab uses owned temporary 127.0.0.1 loopback servers, fictional pages and disposable files; it never scans production feeds or opens the working ApplyDjinn database.

Preserve status evidence by catching HTTPError before its URLError superclass.