Complete worked example
python
from io import StringIO
import math
import pandas as pd
def inspect_example(csv_text):
rows = measured = max_chunk = 0
total = 0.0
with pd.read_csv(StringIO(csv_text), dtype={"id": "string", "score": "Float64"}, keep_default_na=False, na_values={"score": [""]}, chunksize=2) as chunks:
for chunk in chunks:
if list(chunk.columns) != ["id", "score"] or chunk["id"].isna().any() or chunk["id"].str.strip().eq("").any():
raise ValueError("invalid CSV identity or schema")
observed = chunk["score"].dropna()
if not all(math.isfinite(float(v)) for v in observed):
raise ValueError("finite measurements required")
rows += len(chunk)
measured += int(observed.count())
total += float(observed.sum())
max_chunk = max(max_chunk, len(chunk))
return {"rows": rows, "measured": measured, "mean": total / measured if measured else None, "max_chunk": max_chunk}- Reasoning
- Call read_csv with chunksize=2, explicit string ID and nullable Float64 score dtypes, keep_default_na=False and a score-only empty-token rule. Enter the TextFileReader context so its resources close. Validate nonblank IDs and finite observed scores inside each chunk. Add the sum and observed count to small accumulators and track the largest parsed chunk length. Return rows, measured, mean and max_chunk together. For scores two, missing and four, the row count is three, observed count is two and mean is three. The blank score does not become a zero observation. The ID NA remains an ordinary nonblank text identifier.