← pandas: reliable tabular analytics Progress & review ↗

Typorithm · medium

Concatenating batches with schema and identity guarantees

pythonpandasnumerical-computingindependent-practice

Saved on this device

Full lesson · 4 learning steps Open any exercise for focused practice

Step 1 of 4 · intro card

Concatenating batches with schema and identity guarantees

Concept and contract

Combine rows once and distinguish index resetting from deduplication.

Concatenation stacks batches; merging relates rows through keys. Choosing the wrong operation can create extra columns, missing values or unintended row multiplication. A default concat can union incompatible columns, so validate a shared schema before combining independent batches. Resetting the presentation index prevents repeated row ordinals, but it does not establish unique business identity. Check canonical IDs separately after concatenation.

This lesson stacks ordered ID-score batches once, preserves within-batch row order and rejects repeated IDs across the combined population. It is a local in-memory example, not a promise that arbitrarily many batches fit in RAM.

Mechanism and worked example

Construct each batch with the same declared columns and nullable dtypes.

Construct each batch with the same declared columns and nullable dtypes. Reject records with missing or extra fields before construction. Pass the list of frames to one pd.concat call with ignore_index=True, avoiding repeated incremental concatenation. If there are no batches, construct the same typed empty schema explicitly. Reject missing or repeated IDs in the combined frame.

Trace batch one containing a and batch two containing b. Their local row indexes may both start at zero; the combined presentation index becomes zero and one. Their business IDs remain a and b. A repeated a in the second batch is an identity conflict even though the new row index is unique.

Complete worked example

python

import pandas as pd

def inspect_example(batches):
    frames = []
    for batch in batches:
        if any(set(r) != {"id", "score"} for r in batch):
            raise ValueError("batch schema drift")
        frames.append(pd.DataFrame(batch, columns=["id", "score"]).astype({"id": "string", "score": "Float64"}))
    result = pd.concat(frames, ignore_index=True) if frames else pd.DataFrame({"id": pd.Series(dtype="string"), "score": pd.Series(dtype="Float64")})
    if result["id"].isna().any() or result["id"].duplicated().any():
        raise ValueError("duplicate or missing role ID")
    return result

Reasoning: Construct each batch with the same declared columns and nullable dtypes. Reject records with missing or extra fields before construction. Pass the list of frames to one pd.concat call with ignore_index=True, avoiding repeated incremental concatenation. If there are no batches, construct the same typed empty schema explicitly. Reject missing or repeated IDs in the combined frame. Trace batch one containing a and batch two containing b. Their local row indexes may both start at zero; the combined presentation index becomes zero and one. Their business IDs remain a and b. A repeated a in the second batch is an identity conflict even though the new row index is unique.

Open in local Python playground ↗

Focused decision: context is in the complete example

python

result = pd.concat(frames, ignore_index=True)
duplicate_roles = result["id"].duplicated()

Focused decision: Type this short table decision after reading its complete context. The complete imports, table construction, validation, and result checks remain in the worked example. Typing records practice; only the downloadable computer lab executes Python.

Counterexamples and common mistakes

The controlled mistake omits ignore_index=True.

The controlled mistake omits ignore_index=True. Values remain correct but the result contains repeated presentation indexes, violating this output contract. Another mistake assumes ignore_index removes duplicate roles; it only replaces row labels. A third concatenates one new batch into an accumulating frame inside a loop, repeatedly rebuilding larger intermediate tables and creating avoidable work.

Do not silently union unrelated schemas and then interpret the resulting missing cells as absent measurements. If one batch renamed score to ranking, that is a schema change needing explicit mapping, not a reason to create two partially populated columns.

Controlled mistake: inspect the failing table contract

python

import pandas as pd

def inspect_example(batches):
    frames = []
    for batch in batches:
        if any(set(r) != {"id", "score"} for r in batch):
            raise ValueError("batch schema drift")
        frames.append(pd.DataFrame(batch, columns=["id", "score"]).astype({"id": "string", "score": "Float64"}))
    result = pd.concat(frames, ignore_index=False) if frames else pd.DataFrame({"id": pd.Series(dtype="string"), "score": pd.Series(dtype="Float64")})
    if result["id"].isna().any() or result["id"].duplicated().any():
        raise ValueError("duplicate or missing role ID")
    return result

Reasoning: The controlled mistake omits ignore_index=True. Values remain correct but the result contains repeated presentation indexes, violating this output contract. Another mistake assumes ignore_index removes duplicate roles; it only replaces row labels. A third concatenates one new batch into an accumulating frame inside a loop, repeatedly rebuilding larger intermediate tables and creating avoidable work. Do not silently union unrelated schemas and then interpret the resulting missing cells as absent measurements. If one batch renamed score to ranking, that is a schema change needing explicit mapping, not a reason to create two partially populated columns.

Open in local Python playground ↗

Independent practice

Add an origin_batch column that records each row's zero-based input batch number.

Add an origin_batch column that records each row's zero-based input batch number. Preserve one concat call, declared source schema, combined identity validation and the fresh presentation index. Use assign on each small frame before the final combination; the task adds one visible provenance field rather than introducing a relational join.

Predict two rows from the first batch, no rows from an empty middle batch and one row from the third. The retained origin numbers are zero, zero and two. They refer to input batch position, so do not compress away the empty batch when assigning provenance.

Step 1: Attach zero-based input batch provenance without changing role identity or order. Step 2: Retain the working imports, input validation and table schemas. Change the stated policy, run the supplied practice cases, then explain the newly accepted or rejected boundary.

Practice starter: solve before reading the solution

python

import pandas as pd

def inspect_example(batches):
    frames = []
    for batch in batches:
        if any(set(r) != {"id", "score"} for r in batch):
            raise ValueError("batch schema drift")
        frames.append(pd.DataFrame(batch, columns=["id", "score"]).astype({"id": "string", "score": "Float64"}))
    result = pd.concat(frames, ignore_index=True) if frames else pd.DataFrame({"id": pd.Series(dtype="string"), "score": pd.Series(dtype="Float64")})
    if result["id"].isna().any() or result["id"].duplicated().any():
        raise ValueError("duplicate or missing role ID")
    return result

# Independent task: Attach zero-based input batch provenance without changing role identity or order.

Reasoning: Attach zero-based input batch provenance without changing role identity or order.

Open in local Python playground ↗

Hint 1: Find the one field or branch that owns this policy.
Hint 2: Compare the first changed acceptance case with the retained baseline.

Solution, tests and trade-offs

The solution enumerates the input batches and assigns a nullable integer origin_batch column before concatenation.

The solution enumerates the input batches and assigns a nullable integer origin_batch column before concatenation. The no-batch result explicitly includes that column too. Exact frame tests check empty batches, missing scores, duplicate IDs and schema drift, in addition to the new provenance values. Original batches remain unchanged.

An origin number in this teaching fixture is not a durable publisher identifier or a replacement for the canonical role key. In a real importer, use stable source and observation metadata. Preserve the difference between provenance, entity identity and a presentation row index even when all three happen to be represented by small integers in one example.

Practice solution and acceptance checks

python

import pandas as pd

def inspect_example(batches):
    frames = []
    for origin, batch in enumerate(batches):
        if any(set(r) != {"id", "score"} for r in batch):
            raise ValueError("batch schema drift")
        frame = pd.DataFrame(batch, columns=["id", "score"]).astype({"id": "string", "score": "Float64"})
        frames.append(frame.assign(origin_batch=pd.Series(origin, index=frame.index, dtype="Int64")))
    result = pd.concat(frames, ignore_index=True) if frames else pd.DataFrame({"id": pd.Series(dtype="string"), "score": pd.Series(dtype="Float64"), "origin_batch": pd.Series(dtype="Int64")})
    if result["id"].isna().any() or result["id"].duplicated().any():
        raise ValueError("duplicate or missing role ID")
    return result

Reasoning: The solution enumerates the input batches and assigns a nullable integer origin_batch column before concatenation. The no-batch result explicitly includes that column too. Exact frame tests check empty batches, missing scores, duplicate IDs and schema drift, in addition to the new provenance values. Original batches remain unchanged. An origin number in this teaching fixture is not a durable publisher identifier or a replacement for the canonical role key. In a real importer, use stable source and observation metadata. Preserve the difference between provenance, entity identity and a presentation row index even when all three happen to be represented by small integers in one example.

Open in local Python playground ↗

Interview reasoning and next step

Explain concat versus merge with concrete output shapes.

Explain concat versus merge with concrete output shapes. Describe what ignore_index changes and what it leaves untouched. Show why duplicate IDs need a separate check and why collecting frames before one concat is preferable to repeatedly rebuilding an accumulator. State the memory limitation of an in-memory stack.

On the PC, type the one-shot concatenation and recall the index argument. In the app, reason about identity and provenance. Next, stream an owned CSV in chunks with explicit text IDs, retaining only a small aggregate instead of materializing every chunk into a final giant table.

Optional stretch — after the core task works: Add one independently reasoned boundary case and explain its expected value before running it.

Computer practice: https://applydjinn.com/typorithm/pandas-lab-20261106.zip. Install pinned dependencies in an isolated Python environment. Run verify.py, then change the working exercise baseline and run --practice pd-concat. The real pandas tables, exact schemas, unchanged inputs, CSV parser options and owned in-memory SQL reconciliation use owned fixtures only. No production database, feed, account or paid service is used. This lab executes Python; the PC typing trainer and native reading progress do not execute it.

Combine rows once and distinguish index resetting from deduplication.