← pandas: reliable tabular analytics Progress & review ↗

Typorithm · easy

Nullable text and explicit string normalization

pythonpandasnumerical-computingindependent-practice

Saved on this device

Full lesson · 4 learning steps Open any exercise for focused practice

Step 1 of 4 · intro card

Nullable text and explicit string normalization

Concept and contract

Normalize country text without inventing strings for missing input.

Text normalization is a policy, not just a formatting convenience. Stripping whitespace and applying one case convention can make country codes comparable, but it cannot prove that two company names represent the same business. Pandas 3 introduces an inferred str dtype with missing-value behavior that differs from the explicit nullable string dtype. Choose the representation whose missing semantics your contract needs rather than depending on object inference.

This lesson uses dtype="string" for normalized country text. Missing input remains missing. A separate flag checks for a literal hyphen in the supplied text, with unknown text explicitly treated as not matching that flag.

Mechanism and worked example

Create a nullable text Series, then chain str.strip and str.upper to produce a canonical display code.

Create a nullable text Series, then chain str.strip and str.upper to produce a canonical display code. Use str.contains with regex=False when the search term is literal punctuation. Set na=False only for the output flag, preserving missing values in the normalized text itself. Build the two-column result from aligned Series with matching row indexes.

Trace whitespace around ua, a missing entry and the value il-central. Normalized values become UA, missing and IL-CENTRAL. The hyphen flag is true only for the last row. The normalizer does not decide whether a resulting code is an accepted country; an allowlist belongs to a later selection rule.

Complete worked example

python

import pandas as pd

def inspect_example(values):
    text = pd.Series(values, dtype="string")
    country = text.str.strip().str.upper()
    has_hyphen = text.str.contains("-", regex=False, na=False).astype("boolean")
    return pd.DataFrame({"country": country, "has_hyphen": has_hyphen})

Reasoning: Create a nullable text Series, then chain str.strip and str.upper to produce a canonical display code. Use str.contains with regex=False when the search term is literal punctuation. Set na=False only for the output flag, preserving missing values in the normalized text itself. Build the two-column result from aligned Series with matching row indexes. Trace whitespace around ua, a missing entry and the value il-central. Normalized values become UA, missing and IL-CENTRAL. The hyphen flag is true only for the last row. The normalizer does not decide whether a resulting code is an accepted country; an allowlist belongs to a later selection rule.

Open in local Python playground ↗

Focused decision: context is in the complete example

python

text = pd.Series(values, dtype="string")
country = text.str.strip().str.upper()
has_hyphen = text.str.contains("-", regex=False, na=False)

Focused decision: Type this short table decision after reading its complete context. The complete imports, table construction, validation, and result checks remain in the worked example. Typing records practice; only the downloadable computer lab executes Python.

Counterexamples and common mistakes

The controlled mistake fills missing text with an empty string before normalization.

The controlled mistake fills missing text with an empty string before normalization. That turns unavailable data into a present blank value and breaks the missing-value contract. Calling astype(str) as a universal cleaner is also unsafe across versions and dtype choices: explicit missing-preserving conversion is easier to reason about. Another mistake is leaving regex enabled for a literal token containing punctuation, which can accidentally change what matches.

Do not normalize opaque case-sensitive role IDs with the same policy used for country codes. A transformation that is harmless for an ISO-style code can merge two distinct external identifiers.

Controlled mistake: inspect the failing table contract

python

import pandas as pd

def inspect_example(values):
    text = pd.Series(values, dtype="string").fillna("")
    country = text.str.strip().str.upper()
    has_hyphen = text.str.contains("-", regex=False, na=False).astype("boolean")
    return pd.DataFrame({"country": country, "has_hyphen": has_hyphen})

Reasoning: The controlled mistake fills missing text with an empty string before normalization. That turns unavailable data into a present blank value and breaks the missing-value contract. Calling astype(str) as a universal cleaner is also unsafe across versions and dtype choices: explicit missing-preserving conversion is easier to reason about. Another mistake is leaving regex enabled for a literal token containing punctuation, which can accidentally change what matches. Do not normalize opaque case-sensitive role IDs with the same policy used for country codes. A transformation that is harmless for an ISO-style code can merge two distinct external identifiers.

Open in local Python playground ↗

Independent practice

Change only the canonical case convention from uppercase to lowercase.

Change only the canonical case convention from uppercase to lowercase. Preserve stripping, nullable text and literal hyphen detection. This small task practices the vectorized string namespace without introducing regular expressions or a complicated parsing grammar.

Predict ua with whitespace, an already lowercase value and a missing entry. The first two should produce the same normalized text, while the missing entry must remain missing. The hyphen flag does not change because the task changes case rather than punctuation. Keep that unrelated output stable so acceptance tests distinguish a focused policy edit from a wholesale rewrite.

Step 1: Normalize to lowercase while preserving missing text and literal hyphen flags. Step 2: Retain the working imports, input validation and table schemas. Change the stated policy, run the supplied practice cases, then explain the newly accepted or rejected boundary.

Practice starter: solve before reading the solution

python

import pandas as pd

def inspect_example(values):
    text = pd.Series(values, dtype="string")
    country = text.str.strip().str.upper()
    has_hyphen = text.str.contains("-", regex=False, na=False).astype("boolean")
    return pd.DataFrame({"country": country, "has_hyphen": has_hyphen})

# Independent task: Normalize to lowercase while preserving missing text and literal hyphen flags.

Reasoning: Normalize to lowercase while preserving missing text and literal hyphen flags.

Open in local Python playground ↗

Hint 1: Find the one field or branch that owns this policy.
Hint 2: Compare the first changed acceptance case with the retained baseline.

Solution, tests and trade-offs

The solution changes str.upper to str.lower and leaves the flag computation unchanged.

The solution changes str.upper to str.lower and leaves the flag computation unchanged. The lab checks the string extension dtype, missing values, boolean flag and row order. An empty input list must still return both typed output columns. Literal punctuation and an unavailable value are included because happy-path alphabetic strings cannot reveal the controlled defect.

Normalization should be reproducible and documented. In a real importer, retain the original value or a provenance record separately when it is needed for diagnosis. This small teaching function returns only the normalized view and flag; it does not claim to be a full international address parser or a canonical company registry.

Practice solution and acceptance checks

python

import pandas as pd

def inspect_example(values):
    text = pd.Series(values, dtype="string")
    country = text.str.strip().str.lower()
    has_hyphen = text.str.contains("-", regex=False, na=False).astype("boolean")
    return pd.DataFrame({"country": country, "has_hyphen": has_hyphen})

Reasoning: The solution changes str.upper to str.lower and leaves the flag computation unchanged. The lab checks the string extension dtype, missing values, boolean flag and row order. An empty input list must still return both typed output columns. Literal punctuation and an unavailable value are included because happy-path alphabetic strings cannot reveal the controlled defect. Normalization should be reproducible and documented. In a real importer, retain the original value or a provenance record separately when it is needed for diagnosis. This small teaching function returns only the normalized view and flag; it does not claim to be a full international address parser or a canonical company registry.

Open in local Python playground ↗

Interview reasoning and next step

Explain inferred str versus the explicit nullable string choice used here, and explain why isna is preferable to comparing printed sentinels.

Explain inferred str versus the explicit nullable string choice used here, and explain why isna is preferable to comparing printed sentinels. Show when regex=False communicates a more precise contract than the default pattern interpretation. Describe why country normalization, company matching and external job identity require different policies.

On the PC, type the short nullable string pipeline and recall the strip-and-case expression. In the app, answer which entries remain missing. Next, establish canonical role keys with explicit validation so text cleanup does not become an excuse for merging ambiguous identities.

Optional stretch — after the core task works: Add one independently reasoned boundary case and explain its expected value before running it.

Computer practice: https://applydjinn.com/typorithm/pandas-lab-20261106.zip. Install pinned dependencies in an isolated Python environment. Run verify.py, then change the working exercise baseline and run --practice pd-strings. The real pandas tables, exact schemas, unchanged inputs, CSV parser options and owned in-memory SQL reconciliation use owned fixtures only. No production database, feed, account or paid service is used. This lab executes Python; the PC typing trainer and native reading progress do not execute it.

Normalize country text without inventing strings for missing input.