Pagination without cycles or partial-success claims

pythonapi-clientsautomationindependent-practice

Saved on this device

Full lesson · 4 learning steps Open any exercise for focused practice

Step 1 of 4 · intro card

Pagination without cycles or partial-success claims

Concept and contract

Follow an injected page provider while bounding work and distinguishing a complete collection from a truncated one.

Pagination is a state machine, not a request repeated until something looks empty. A client must know its initial cursor, how a page declares completion and which bounds limit the work.

This lesson receives a callable page provider, separating page traversal from HTTP transport. The lab injects a small in-memory provider with recorded calls. No real API or background scan is needed to test cursor and collection behavior.

Mechanism and worked example

The function begins with None as its initial cursor, checks for a previously visited cursor and calls the provider at most the declared page budget.

The function begins with None as its initial cursor, checks for a previously visited cursor and calls the provider at most the declared page budget. Each page must contain an items list and an explicit next field.

A next value of None completes the collection; a nonempty string advances it. Identifiers are deduplicated and sorted only after completion, so callers receive a reproducible complete result under the stated identity rule.

Complete worked example

python

def collect_pages(fetch_page, max_pages=3):
    if type(max_pages) is not int or not 1 <= max_pages <= 5:
        raise ValueError("page budget must be 1..5")
    cursor, seen, ids = None, set(), []
    for _ in range(max_pages):
        if cursor in seen:
            raise ValueError("cursor cycle")
        seen.add(cursor)
        page = fetch_page(cursor)
        if not isinstance(page, dict) or "next" not in page or not isinstance(page.get("items"), list):
            raise ValueError("page shape required")
        if any(not isinstance(item, str) or not item for item in page["items"]):
            raise ValueError("nonempty identifiers required")
        ids.extend(page["items"])
        cursor = page["next"]
        if cursor is None:
            return sorted(set(ids))
        if not isinstance(cursor, str) or not cursor:
            raise ValueError("cursor must be nonempty text or None")
    raise ValueError("page budget exhausted before completion")

Reasoning: Follow an injected page provider while bounding work and distinguishing a complete collection from a truncated one. Compare the stated values and failure boundary before changing the implementation.

Counterexamples and common mistakes

The counterexample returns whatever it has after the page budget expires.

The counterexample returns whatever it has after the page budget expires. The output can look plausible while omitting later pages, and a caller may incorrectly interpret it as a complete market snapshot.

An empty items list is not necessarily completion if next still points onward. Detect repeated cursors instead of relying on an endpoint eventually stopping. Missing next is a shape error under this contract, not an implicit successful ending.

Counterexample: a runnable policy defect

python

def collect_pages(fetch_page, max_pages=3):
    if type(max_pages) is not int or not 1 <= max_pages <= 5:
        raise ValueError("page budget must be 1..5")
    cursor, seen, ids = None, set(), []
    for _ in range(max_pages):
        if cursor in seen:
            raise ValueError("cursor cycle")
        seen.add(cursor)
        page = fetch_page(cursor)
        if not isinstance(page, dict) or "next" not in page or not isinstance(page.get("items"), list):
            raise ValueError("page shape required")
        if any(not isinstance(item, str) or not item for item in page["items"]):
            raise ValueError("nonempty identifiers required")
        ids.extend(page["items"])
        cursor = page["next"]
        if cursor is None:
            return sorted(set(ids))
        if not isinstance(cursor, str) or not cursor:
            raise ValueError("cursor must be nonempty text or None")
    return sorted(set(ids))

Reasoning: This runnable counterexample violates the declared contract; the companion lab requires it to fail.

Independent practice: API client task

Add a max_items parameter defaulting to ten and cap the number of distinct collected identifiers.

Add a max_items parameter defaulting to ten and cap the number of distinct collected identifiers. Duplicate identifiers do not consume an additional distinct-item allowance.

Check the bound after each page before following another cursor. Start with a complete two-page result at its exact limit, then reduce the limit and require an explicit failure. Keep the existing page count and cycle checks intact.

Step 1: Add a hard budget for distinct collected identifiers. Step 2: Reject an incomplete or overflowing collection instead of returning partial success.

Practice starter: solve before reading the solution

python

def collect_pages(fetch_page, max_pages=3):
    # Remove this placeholder, then adapt the retained baseline:
    # Add a hard budget for distinct collected identifiers.
    raise NotImplementedError
    if type(max_pages) is not int or not 1 <= max_pages <= 5:
        raise ValueError("page budget must be 1..5")
    cursor, seen, ids = None, set(), []
    for _ in range(max_pages):
        if cursor in seen:
            raise ValueError("cursor cycle")
        seen.add(cursor)
        page = fetch_page(cursor)
        if not isinstance(page, dict) or "next" not in page or not isinstance(page.get("items"), list):
            raise ValueError("page shape required")
        if any(not isinstance(item, str) or not item for item in page["items"]):
            raise ValueError("nonempty identifiers required")
        ids.extend(page["items"])
        cursor = page["next"]
        if cursor is None:
            return sorted(set(ids))
        if not isinstance(cursor, str) or not cursor:
            raise ValueError("cursor must be nonempty text or None")
    raise ValueError("page budget exhausted before completion")

Reasoning: Add a hard budget for distinct collected identifiers. Reject an incomplete or overflowing collection instead of returning partial success.

Hint 1: Count set(ids), not the number of repeated appearances.
Hint 2: Check the new bound immediately after extending the current page.

Solution, tests and trade-offs

The companion runner adapts the case dictionary into a callable and records the cursor sequence.

The companion runner adapts the case dictionary into a callable and records the cursor sequence. It checks that a repeated cursor is rejected before another fetch and that an exhausted page budget never becomes a partial success.

Practice adds exact-fit and overflow item counts. The provider fixtures are visible source data rather than hidden network state, and a provider exception propagates instead of silently producing an empty collection.

Practice solution and acceptance checks

python

def collect_pages(fetch_page, max_pages=3, max_items=10):
    if type(max_pages) is not int or not 1 <= max_pages <= 5:
        raise ValueError("page budget must be 1..5")
    if type(max_items) is not int or not 1 <= max_items <= 20:
        raise ValueError("item budget must be 1..20")
    cursor, seen, ids = None, set(), []
    for _ in range(max_pages):
        if cursor in seen:
            raise ValueError("cursor cycle")
        seen.add(cursor)
        page = fetch_page(cursor)
        if not isinstance(page, dict) or "next" not in page or not isinstance(page.get("items"), list):
            raise ValueError("page shape required")
        if any(not isinstance(item, str) or not item for item in page["items"]):
            raise ValueError("nonempty identifiers required")
        ids.extend(page["items"])
        if len(set(ids)) > max_items:
            raise ValueError("distinct item budget exceeded")
        cursor = page["next"]
        if cursor is None:
            return sorted(set(ids))
        if not isinstance(cursor, str) or not cursor:
            raise ValueError("cursor must be nonempty text or None")
    raise ValueError("page budget exhausted before completion")

Reasoning: Count set(ids), not the number of repeated appearances. Check the new bound immediately after extending the current page.

Interview reasoning and next step

Explain how transport, traversal and publication fit together.

Explain how transport, traversal and publication fit together. A retried request should repeat the same cursor, while committing a successful page advances it once.

A real source may also require snapshot tokens, stable ordering and duplicate-version rules. This lesson assumes one declared page shape and identity format. The later capstone combines bounded HTTP and traversal while preserving the requirement that incomplete work must not be advertised as a complete report.

Optional stretch — after the core task works: Design a resumable state record that preserves the last committed cursor.

Computer practice: https://applydjinn.com/typorithm/python-automation-lab-20261105.zip. Run verify.py using your selected Python interpreter, then adapt the retained exercise baseline and check it with --practice followed by the lesson id. The standard-library lab uses owned temporary 127.0.0.1 loopback servers, fictional pages and disposable files; it never scans production feeds or opens the working ApplyDjinn database.

Follow an injected page provider while bounding work and distinguishing a complete collection from a truncated one.