← NumPy: arrays and numerical computing Progress & review ↗

Typorithm · medium

Reference-only scaling and held-out data

pythonnumpynumerical-computingindependent-practice

Saved on this device

Full lesson · 4 learning steps Open any exercise for focused practice

Step 1 of 4 · intro card

Reference-only scaling and held-out data

Concept and contract

Estimate feature statistics from the reference matrix and reuse them without fitting to the evaluation population.

A numerical transformation can leak information even without an explicit predictive model. If held-out observations influence the mean or scale used to transform themselves, the evaluation no longer represents a transformation estimated independently from those observations. This lesson computes feature means and population standard deviations from a declared reference matrix only, then applies them to another aligned matrix.

The function does not create a train/test split or claim that the reference population is representative. The caller supplies the populations deliberately. Keeping those roles separate now prepares for later statistics and ML courses, where split design, time ordering and leakage need deeper treatment.

Mechanism and worked example

The worked example validates non-empty finite reference rows and the same number of feature columns in the evaluation rows.

The worked example validates non-empty finite reference rows and the same number of feature columns in the evaluation rows. It computes mean and std with axis=0 on reference. A zero standard deviation is replaced with one as an explicit numerical policy, avoiding division by zero while leaving differences on that constant feature visible. The resulting transformation uses column-wise broadcasting already introduced earlier in the course.

ddof=0 here describes a population-standard-deviation calculation on the supplied reference. It is not a universal unbiased estimator claim. A different statistical estimator would require different sample-size assumptions, especially for a reference containing only one row.

Complete worked example

python

import numpy as np

def inspect_example(reference_rows, evaluation_rows):
    reference = np.asarray(reference_rows, dtype=np.float64)
    if reference.ndim != 2 or 0 in reference.shape or not np.isfinite(reference).all():
        raise ValueError("expected a non-empty finite reference matrix")
    samples = (np.asarray(evaluation_rows, dtype=np.float64) if len(evaluation_rows)
               else np.empty((0, reference.shape[1]), dtype=np.float64))
    if samples.ndim != 2 or samples.shape[1] != reference.shape[1] or not np.isfinite(samples).all():
        raise ValueError("evaluation features must match the reference")
    mean = reference.mean(axis=0)
    scale = reference.std(axis=0, ddof=0)
    scale = np.where(scale == 0, 1.0, scale)
    result = (samples - mean) / scale
    return result

Reasoning: The worked example validates non-empty finite reference rows and the same number of feature columns in the evaluation rows. It computes mean and std with axis=0 on reference. A zero standard deviation is replaced with one as an explicit numerical policy, avoiding division by zero while leaving differences on that constant feature visible. The resulting transformation uses column-wise broadcasting already introduced earlier in the course. ddof=0 here describes a population-standard-deviation calculation on the supplied reference. It is not a universal unbiased estimator claim. A different statistical estimator would require different sample-size assumptions, especially for a reference containing only one row.

Open in local Python playground ↗

Focused decision: context is in the complete example

python

mean = reference.mean(axis=0)
scale = reference.std(axis=0, ddof=0)
scale = np.where(scale == 0, 1.0, scale)
result = (samples - mean) / scale

Focused decision: Type this short numerical decision after reading its complete context. The complete imports, array construction, validation, and result checks remain in the worked example. Typing records practice; only the downloadable computer lab executes Python.

Counterexamples and common mistakes

The controlled counterexample includes held-out rows when calculating the mean.

The controlled counterexample includes held-out rows when calculating the mean. Extreme evaluation values then shift the transformation parameters, making the held-out data participate in its own reference. Output may still be finite and shaped correctly, so shape checks alone are insufficient. Another error is computing row means, which normalizes observations internally rather than reusing per-feature reference statistics.

A constant reference feature also requires an explicit policy. Dropping it, rejecting it or preserving a guarded difference can each be appropriate for a different consumer. This fixture declares a scale of one and tests that choice rather than hiding it in a blanket warning suppression.

Controlled mistake: inspect the failing numerical contract

python

import numpy as np

def inspect_example(reference_rows, evaluation_rows):
    reference = np.asarray(reference_rows, dtype=np.float64)
    if reference.ndim != 2 or 0 in reference.shape or not np.isfinite(reference).all():
        raise ValueError("expected a non-empty finite reference matrix")
    samples = (np.asarray(evaluation_rows, dtype=np.float64) if len(evaluation_rows)
               else np.empty((0, reference.shape[1]), dtype=np.float64))
    if samples.ndim != 2 or samples.shape[1] != reference.shape[1] or not np.isfinite(samples).all():
        raise ValueError("evaluation features must match the reference")
    mean = np.concatenate((reference, samples), axis=0).mean(axis=0)
    scale = reference.std(axis=0, ddof=0)
    scale = np.where(scale == 0, 1.0, scale)
    result = (samples - mean) / scale
    return result

Reasoning: The controlled counterexample includes held-out rows when calculating the mean. Extreme evaluation values then shift the transformation parameters, making the held-out data participate in its own reference. Output may still be finite and shaped correctly, so shape checks alone are insufficient. Another error is computing row means, which normalizes observations internally rather than reusing per-feature reference statistics. A constant reference feature also requires an explicit policy. Dropping it, rejecting it or preserving a guarded difference can each be appropriate for a different consumer. This fixture declares a scale of one and tests that choice rather than hiding it in a blanket warning suppression.

Open in local Python playground ↗

Independent practice

Keep the reference-only mean and scale, but clip transformed results to the interval [-2, 2].

Keep the reference-only mean and scale, but clip transformed results to the interval [-2, 2]. The small task adds a bounded output policy after normalization; it must not clip the raw reference before fitting or recompute statistics from evaluation data.

Predict the result for a held-out value far above the reference mean. The changed output should stop at two, while ordinary in-range values remain unchanged. Include the constant-feature column and an empty evaluation matrix so the policy remains consistent across shapes and boundary populations.

Step 1: Clip standardized evaluation values to [-2, 2] without refitting the transformation. Step 2: Retain the working imports, input validation and array shapes. Change the stated policy, run the supplied practice cases, then explain the newly accepted or rejected boundary.

Practice starter: solve before reading the solution

python

import numpy as np

def inspect_example(reference_rows, evaluation_rows):
    reference = np.asarray(reference_rows, dtype=np.float64)
    if reference.ndim != 2 or 0 in reference.shape or not np.isfinite(reference).all():
        raise ValueError("expected a non-empty finite reference matrix")
    samples = (np.asarray(evaluation_rows, dtype=np.float64) if len(evaluation_rows)
               else np.empty((0, reference.shape[1]), dtype=np.float64))
    if samples.ndim != 2 or samples.shape[1] != reference.shape[1] or not np.isfinite(samples).all():
        raise ValueError("evaluation features must match the reference")
    mean = reference.mean(axis=0)
    scale = reference.std(axis=0, ddof=0)
    scale = np.where(scale == 0, 1.0, scale)
    result = (samples - mean) / scale
    return result

# Independent task: Clip standardized evaluation values to [-2, 2] without refitting the transformation.

Reasoning: Clip standardized evaluation values to [-2, 2] without refitting the transformation.

Open in local Python playground ↗

Hint 1: Do not change the population used for mean or scale.
Hint 2: Apply np.clip(result, -2.0, 2.0) at the return boundary.

Solution, tests and trade-offs

The solution applies np.clip to the already standardized matrix.

The solution applies np.clip to the already standardized matrix. Tests compare both an extreme held-out value and a value exactly at the boundary. Empty evaluation input returns an empty float64 matrix with the known reference feature count, while empty reference input remains invalid because no transformation parameters can be estimated.

Clipping changes numerical information and may conceal the magnitude of a distribution shift. It is a stated exercise policy, not a recommendation to improve every model. A real pipeline should record and evaluate such preprocessing decisions using the same fitted parameters and carefully separated populations.

Practice solution and acceptance checks

python

import numpy as np

def inspect_example(reference_rows, evaluation_rows):
    reference = np.asarray(reference_rows, dtype=np.float64)
    if reference.ndim != 2 or 0 in reference.shape or not np.isfinite(reference).all():
        raise ValueError("expected a non-empty finite reference matrix")
    samples = (np.asarray(evaluation_rows, dtype=np.float64) if len(evaluation_rows)
               else np.empty((0, reference.shape[1]), dtype=np.float64))
    if samples.ndim != 2 or samples.shape[1] != reference.shape[1] or not np.isfinite(samples).all():
        raise ValueError("evaluation features must match the reference")
    mean = reference.mean(axis=0)
    scale = reference.std(axis=0, ddof=0)
    scale = np.where(scale == 0, 1.0, scale)
    result = (samples - mean) / scale
    return np.clip(result, -2.0, 2.0)

Reasoning: The solution applies np.clip to the already standardized matrix. Tests compare both an extreme held-out value and a value exactly at the boundary. Empty evaluation input returns an empty float64 matrix with the known reference feature count, while empty reference input remains invalid because no transformation parameters can be estimated. Clipping changes numerical information and may conceal the magnitude of a distribution shift. It is a stated exercise policy, not a recommendation to improve every model. A real pipeline should record and evaluate such preprocessing decisions using the same fitted parameters and carefully separated populations.

Open in local Python playground ↗

Interview reasoning and next step

Explain which rows are allowed to influence a fitted transformation, why axis=0 is used, and how the chosen zero-variance policy affects a held-out observation.

Explain which rows are allowed to influence a fitted transformation, why axis=0 is used, and how the chosen zero-variance policy affects a held-out observation. Describe how a shape-correct finite output can still leak information. Distinguish this numerical implementation from the broader question of whether a particular scaling strategy is statistically suitable.

Type the short reference-statistics and broadcast sequence, then recall the order of fit, transform and optional clipping. App questions focus on population ownership. The final capstone will combine validation, selection, canonical identities, ranking and an honest report denominator.

Optional stretch — after the core task works: Add one independently reasoned boundary case and explain its expected value before running it.

Computer practice: https://applydjinn.com/typorithm/numpy-lab-20261106.zip. Install pinned dependencies in an isolated Python environment. Run verify.py, then change the working exercise baseline and run --practice np-scaling. The real NumPy arrays, numerical contracts, random state checks and temporary numeric files use owned fixtures only. No production database, feed, account or paid service is used. This lab executes Python; the PC typing trainer and native reading progress do not execute it.

Estimate feature statistics from the reference matrix and reuse them without fitting to the evaluation population.