Typorithm · medium

Iterators, generators and consumption boundaries

pythonscriptingdata-boundariesindependent-practice
Saved on this device
Full lesson · 4 learning steps Open any exercise for focused practice
1

Step 1 of 4 · intro card

Iterators, generators and consumption boundaries

0%

Concept and contract

Distinguish a reusable iterable from a one-pass iterator and observe when work actually occurs.

A generator yields values as it is consumed. The worked titles function does not build a list of every title before its caller asks for one. take_titles creates one iterator, consumes a bounded prefix through islice and then collects the remainder of this small finite teaching fixture.

Constructing the generator is different from advancing it. The lab uses a recording input iterator to verify that construction consumes nothing and the first next call consumes only the first record. This establishes a concrete laziness property rather than inferring it from the presence of a generator keyword.

Mechanism and worked example

An iterable can supply an iterator, while an iterator carries its current position.

An iterable can supply an iterator, while an iterator carries its current position. islice consumes from that current position. After the prefix is collected, list(stream) continues from the remaining position; it does not restart the source at its first title.

The worked remainder collection intentionally materializes the rest of a finite fixture. It is not a constant-memory claim for an unbounded feed. A real infinite or very large stream needs bounded downstream consumption too, even if its initial transformation is lazy.

Complete worked example

python

from itertools import islice

def titles(rows):
    for row in rows:
        yield row["title"]

def take_titles(rows, count):
    stream = titles(rows)
    prefix = list(islice(stream, count))
    return {"prefix": prefix, "rest": list(stream)}
Reasoning
The same iterator carries its position from the prefix into the remainder.

Counterexamples and common mistakes

The counterexample constructs a new titles iterator when collecting the remainder, so it repeats the complete input instead of returning what remains.

The counterexample constructs a new titles iterator when collecting the remainder, so it repeats the complete input instead of returning what remains. A different mistake calls list(stream) twice and expects both calls to see the same values after the first has exhausted the iterator.

If several consumers need independent traversals, choose a deliberate replay, buffering or source-reopening policy. Do not assume all iterables can be restarted cheaply or at all. A file, network stream and in-memory list have different lifetime and consistency properties.

Counterexample: a plausible but incorrect implementation

python

from itertools import islice

def titles(rows):
    for row in rows:
        yield row["title"]

def take_titles(rows, count):
    prefix = list(islice(titles(rows), count))
    return {"prefix": prefix, "rest": list(titles(rows))}
Reasoning
Creating a fresh iterator restarts the input instead of continuing it.

Independent practice: small Python task

Return two consecutive prefixes of the requested length from the same stream.

Return two consecutive prefixes of the requested length from the same stream. Call the result fields first and second. For three titles and a count of one, predict A then B; do not restart the iterator or collect the entire remaining input.

Create the iterator once, apply islice to it twice and materialize only each requested prefix. The count is a nonnegative integer under this task's contract. Zero consumes no records, and requesting more than the finite population simply returns the available values.

Step 1: Create the title iterator once. Step 2: Consume two islice prefixes from that same object.

Practice starter: solve before reading the solution

python

from itertools import islice

def titles(rows):
    for row in rows:
        yield row["title"]

def take_titles(rows, count):
    # Return first and second consecutive bounded prefixes.
    raise NotImplementedError
Reasoning
Take both pieces from one stream without materializing the remaining population.
  • Hint 1: Do not call titles(rows) again for the second prefix.
  • Hint 2: A zero slice should consume no items.

Solution, tests and trade-offs

The solution uses one generator instance for both prefix operations.

The solution uses one generator instance for both prefix operations. A count of one yields A in first and B in second; a count larger than the fixture yields all titles first and an empty second list. Source rows remain unchanged throughout the consumption.

Acceptance checks sequential position and the recording laziness probe separately. A function can preserve values on a list fixture while eagerly consuming an iterator at construction time, so both properties matter when moving from toy collections to incremental processing.

Practice solution and acceptance checks

python

from itertools import islice

def titles(rows):
    for row in rows:
        yield row["title"]

def take_titles(rows, count):
    stream = titles(rows)
    first = list(islice(stream, count))
    second = list(islice(stream, count))
    return {"first": first, "second": second}
Reasoning
Consecutive slices share iterator state; a separate probe checks construction laziness.

Interview reasoning and next step

In an interview, explain where an iterator's state lives and which operation consumes it.

In an interview, explain where an iterator's state lives and which operation consumes it. Demonstrate construction, one next call and a subsequent bounded slice. Avoid claiming that every generator makes the entire application streaming when a downstream sort or list still materializes everything.

The next lesson introduces small data objects. It keeps runtime validation separate from type annotations and uses a flat frozen record without claiming that every nested object has become deeply immutable.

Optional stretch — after the core task works: Pass a side-effect-recording source and verify that neither construction nor a zero slice reads a record.

Computer practice: https://applydjinn.com/typorithm/python-lab-20261105.zip. Run verify.py with your selected Python interpreter, then complete an exercise and check it with --practice followed by the lesson id. The lab uses only the standard library and disposable local fixtures; it never connects to the production database or company feeds.

Distinguish a reusable iterable from a one-pass iterator and observe when work actually occurs.