Conditional requests with URL-scoped ETags

pythonapi-clientsautomationindependent-practice

Saved on this device

Full lesson · 4 learning steps Open any exercise for focused practice

Step 1 of 4 · intro card

Conditional requests with URL-scoped ETags

Concept and contract

Reuse a cached representation only when its URL and validator belong to the requested resource.

An ETag is an opaque validator for a representation, not a globally unique identity for every resource on a server. A cache entry must retain the requested URL alongside its validator and data.

This lesson sends If-None-Match only for an entry belonging to the same URL. A 304 response then means reuse that stored representation; it does not contain a new JSON document to parse as an empty or fresh result.

Mechanism and worked example

The function constructs request headers from a matching cache entry, disables automatic redirects and reads a changed response within a byte bound.

The function constructs request headers from a matching cache entry, disables automatic redirects and reads a changed response within a byte bound. urllib reports 304 through HTTPError, so the code closes that response and reuses the matching entry only in that branch.

A new 200 response records its URL, received ETag and parsed object together. The worked helper assumes a cache entry previously created by this code; the practice strengthens that input boundary.

Complete worked example

python

from urllib.request import build_opener, HTTPRedirectHandler, ProxyHandler

class NoRedirect(HTTPRedirectHandler):
    def redirect_request(self, request, file, code, message, headers, new_url):
        return None

import json
from urllib.request import Request
from urllib.error import HTTPError

def invalid_constant(value):
    raise ValueError("non-finite JSON constant")

def cached_document(url, cached=None):
    headers = {}
    if isinstance(cached, dict) and cached.get("url") == url and isinstance(cached.get("etag"), str):
        headers["If-None-Match"] = cached["etag"]
    try:
        with build_opener(ProxyHandler({}), NoRedirect()).open(Request(url, headers=headers), timeout=2) as response:
            if response.headers.get_content_type() != "application/json":
                raise ValueError("JSON media type required")
            raw, etag = response.read(4097), response.headers.get("ETag")
    except HTTPError as error:
        status = error.code
        error.close()
        if status == 304 and "If-None-Match" in headers:
            return dict(cached)
        raise
    if len(raw) > 4096:
        raise ValueError("response exceeds budget")
    data = json.loads(raw.decode("utf-8"), parse_constant=invalid_constant)
    if not isinstance(data, dict):
        raise ValueError("object required")
    return {"url": url, "etag": etag, "data": data}

Reasoning: Reuse a cached representation only when its URL and validator belong to the requested resource. Compare the stated values and failure boundary before changing the implementation.

Counterexamples and common mistakes

The counterexample sends a validator from a different URL.

The counterexample sends a validator from a different URL. Two country requests can share the same textual ETag, so the server may legitimately return 304 while the client reuses the wrong country’s stored data.

A cache key based only on company name or endpoint path loses query identity. Never convert a 304 into an empty dataset, and never invent a validator when a response omitted one.

Counterexample: a runnable policy defect

python

from urllib.request import build_opener, HTTPRedirectHandler, ProxyHandler

class NoRedirect(HTTPRedirectHandler):
    def redirect_request(self, request, file, code, message, headers, new_url):
        return None

import json
from urllib.request import Request
from urllib.error import HTTPError

def invalid_constant(value):
    raise ValueError("non-finite JSON constant")

def cached_document(url, cached=None):
    headers = {}
    if isinstance(cached, dict) and True and isinstance(cached.get("etag"), str):
        headers["If-None-Match"] = cached["etag"]
    try:
        with build_opener(ProxyHandler({}), NoRedirect()).open(Request(url, headers=headers), timeout=2) as response:
            if response.headers.get_content_type() != "application/json":
                raise ValueError("JSON media type required")
            raw, etag = response.read(4097), response.headers.get("ETag")
    except HTTPError as error:
        status = error.code
        error.close()
        if status == 304 and "If-None-Match" in headers:
            return dict(cached)
        raise
    if len(raw) > 4096:
        raise ValueError("response exceeds budget")
    data = json.loads(raw.decode("utf-8"), parse_constant=invalid_constant)
    if not isinstance(data, dict):
        raise ValueError("object required")
    return {"url": url, "etag": etag, "data": data}

Reasoning: This runnable counterexample violates the declared contract; the companion lab requires it to fail.

Independent practice: API client task

Require cached data to be a dictionary before using its ETag for a conditional request.

Require cached data to be a dictionary before using its ETag for a conditional request. When the supplied cache record has a string in data, fetch a fresh response without the conditional header.

Start by checking the recorded request headers in the local fixture. Keep correctly matched entries reusable and preserve the URL, timeout and response-byte policies. This is one additional cache-record validation rule.

Step 1: Reject malformed cached data for conditional reuse. Step 2: Fetch a fresh response without sending its validator when the cached data is not a dict.

Practice starter: solve before reading the solution

python

from urllib.request import build_opener, HTTPRedirectHandler, ProxyHandler

class NoRedirect(HTTPRedirectHandler):
    def redirect_request(self, request, file, code, message, headers, new_url):
        return None

import json
from urllib.request import Request
from urllib.error import HTTPError

def invalid_constant(value):
    raise ValueError("non-finite JSON constant")

def cached_document(url, cached=None):
    # Remove this placeholder, then adapt the retained baseline:
    # Reject malformed cached data for conditional reuse.
    raise NotImplementedError
    headers = {}
    if isinstance(cached, dict) and cached.get("url") == url and isinstance(cached.get("etag"), str):
        headers["If-None-Match"] = cached["etag"]
    try:
        with build_opener(ProxyHandler({}), NoRedirect()).open(Request(url, headers=headers), timeout=2) as response:
            if response.headers.get_content_type() != "application/json":
                raise ValueError("JSON media type required")
            raw, etag = response.read(4097), response.headers.get("ETag")
    except HTTPError as error:
        status = error.code
        error.close()
        if status == 304 and "If-None-Match" in headers:
            return dict(cached)
        raise
    if len(raw) > 4096:
        raise ValueError("response exceeds budget")
    data = json.loads(raw.decode("utf-8"), parse_constant=invalid_constant)
    if not isinstance(data, dict):
        raise ValueError("object required")
    return {"url": url, "etag": etag, "data": data}

Reasoning: Reject malformed cached data for conditional reuse. Fetch a fresh response without sending its validator when the cached data is not a dict.

Hint 1: Extend the cache-entry guard before adding If-None-Match.
Hint 2: Inspect the actual request header as well as the returned object.

Solution, tests and trade-offs

The fixture server returns 304 only when its configured validator matches the received header; otherwise it returns a controlled fresh object.

The fixture server returns 304 only when its configured validator matches the received header; otherwise it returns a controlled fresh object. Cases inspect both the returned value and the actual request header.

They include a mismatched cache URL, a matching validator and an unprompted 304 that must remain an error. Practice adds malformed cached data. Inputs stay unchanged, and old course content or application data is never used as a fixture.

Practice solution and acceptance checks

python

from urllib.request import build_opener, HTTPRedirectHandler, ProxyHandler

class NoRedirect(HTTPRedirectHandler):
    def redirect_request(self, request, file, code, message, headers, new_url):
        return None

import json
from urllib.request import Request
from urllib.error import HTTPError

def invalid_constant(value):
    raise ValueError("non-finite JSON constant")

def cached_document(url, cached=None):
    headers = {}
    if isinstance(cached, dict) and cached.get("url") == url and isinstance(cached.get("etag"), str) and isinstance(cached.get("data"), dict):
        headers["If-None-Match"] = cached["etag"]
    try:
        with build_opener(ProxyHandler({}), NoRedirect()).open(Request(url, headers=headers), timeout=2) as response:
            if response.headers.get_content_type() != "application/json":
                raise ValueError("JSON media type required")
            raw, etag = response.read(4097), response.headers.get("ETag")
    except HTTPError as error:
        status = error.code
        error.close()
        if status == 304 and "If-None-Match" in headers:
            return dict(cached)
        raise
    if len(raw) > 4096:
        raise ValueError("response exceeds budget")
    data = json.loads(raw.decode("utf-8"), parse_constant=invalid_constant)
    if not isinstance(data, dict):
        raise ValueError("object required")
    return {"url": url, "etag": etag, "data": data}

Reasoning: Extend the cache-entry guard before adding If-None-Match. Inspect the actual request header as well as the returned object.

Interview reasoning and next step

Explain that validation freshness, expiry and retention are different cache policies.

Explain that validation freshness, expiry and retention are different cache policies. An ETag revalidation can confirm an unchanged representation without giving the cache infinite freshness or permission to retain sensitive data forever.

A production shared cache may additionally need account and authorization scope plus Vary handling. Our smaller public fictional client demonstrates the URL-validator binding needed before adding a Redis or disk storage adapter.

Optional stretch — after the core task works: Add an explicit account scope and explain how it changes the cache key.

Computer practice: https://applydjinn.com/typorithm/python-automation-lab-20261105.zip. Run verify.py using your selected Python interpreter, then adapt the retained exercise baseline and check it with --practice followed by the lesson id. The standard-library lab uses owned temporary 127.0.0.1 loopback servers, fictional pages and disposable files; it never scans production feeds or opens the working ApplyDjinn database.

Reuse a cached representation only when its URL and validator belong to the requested resource.