PyTorch: confusion counts and imbalanced evaluation

pythontorchindependent-practice

Saved on this device

Full lesson · 4 learning steps Open any exercise for focused practice

Step 1 of 4 · intro card

PyTorch: confusion counts and imbalanced evaluation

Concept and contract

Inspect class-specific failures instead of trusting accuracy alone.

An overall accuracy can hide failure on a rare but important class. Confusion counts make the evaluation population and error types explicit.

For binary class one, true positives are correctly predicted positives and false negatives are missed actual positives. Recall divides true positives by all actual positives, and its undefined-denominator policy should be stated rather than silently invented.

Mechanism and worked example

The helper validates matching nonempty binary integer lists, builds owned tensors and computes accuracy, true positives, false negatives and positive-class recall..

The helper validates matching nonempty binary integer lists, builds owned tensors and computes accuracy, true positives, false negatives and positive-class recall.

If the batch contains no actual positive, recall is None because that class was not observed. This is an explicit reporting choice, not a claim that performance on the absent class is perfect or zero.

Complete worked example

python

from bootstrap_lab import torch_setup
torch = torch_setup()

def binary_metrics(predictions, targets):
    if len(predictions) != len(targets) or not targets:
        raise ValueError('matching nonempty batches required')
    if any(type(v) is not int or v not in (0, 1) for v in predictions + targets):
        raise ValueError('binary integer classes required')
    predicted = torch.tensor(predictions)
    actual = torch.tensor(targets)
    tp = ((predicted == 1) & (actual == 1)).sum().item()
    fn = ((predicted == 0) & (actual == 1)).sum().item()
    return {'accuracy': (predicted == actual).float().mean().item(),
            'recall': tp / (tp + fn) if tp + fn else None,
            'tp': tp, 'fn': fn}

Reasoning: The helper validates matching nonempty binary integer lists, builds owned tensors and computes accuracy, true positives, false negatives and positive-class recall.

Counterexamples and common mistakes

The counterexample divides recall by the number of predicted positives instead of actual positives.

The counterexample divides recall by the number of predicted positives instead of actual positives. That is precision’s denominator and answers a different question.

A high-accuracy all-negative classifier can have zero positive recall on an imbalanced batch. Do not change the metric name to suit a convenient formula, and retain class support when comparing models across different populations.

Counterexample: a runnable framework mistake

python

from bootstrap_lab import torch_setup
torch = torch_setup()

def binary_metrics(predictions, targets):
    if len(predictions) != len(targets) or not targets:
        raise ValueError('matching nonempty batches required')
    if any(type(v) is not int or v not in (0, 1) for v in predictions + targets):
        raise ValueError('binary integer classes required')
    predicted = torch.tensor(predictions)
    actual = torch.tensor(targets)
    tp = ((predicted == 1) & (actual == 1)).sum().item()
    fn = ((predicted == 0) & (actual == 1)).sum().item()
    return {'accuracy': (predicted == actual).float().mean().item(),
            'recall': tp / int((predicted == 1).sum()) if int((predicted == 1).sum()) else None,
            'tp': tp, 'fn': fn}

Reasoning: The counterexample divides recall by the number of predicted positives instead of actual positives. That is precision’s denominator and answers a different question.

Independent practice: one bounded framework change

Add false-positive count and precision with None when there are no predicted positives.

Add false-positive count and precision with None when there are no predicted positives. Preserve recall’s original denominator and undefined policy.

This introduces the complementary error direction using the same predictions and targets. Keep integer counts distinct from floating ratios, and avoid adding a tiny epsilon merely to hide the fact that a denominator is absent.

Step 1: Add false positives and precision with an explicit undefined case. Step 2: Count predicted positives against actual negatives.

Practice starter: solve before reading the solution

python

from bootstrap_lab import torch_setup
torch = torch_setup()

def binary_metrics(predictions, targets):
    # Remove this placeholder, then adapt the retained baseline:
    # Add false positives and precision with an explicit undefined case.
    raise NotImplementedError
    if len(predictions) != len(targets) or not targets:
        raise ValueError('matching nonempty batches required')
    if any(type(v) is not int or v not in (0, 1) for v in predictions + targets):
        raise ValueError('binary integer classes required')
    predicted = torch.tensor(predictions)
    actual = torch.tensor(targets)
    tp = ((predicted == 1) & (actual == 1)).sum().item()
    fn = ((predicted == 0) & (actual == 1)).sum().item()
    return {'accuracy': (predicted == actual).float().mean().item(),
            'recall': tp / (tp + fn) if tp + fn else None,
            'tp': tp, 'fn': fn}

Reasoning: Add false positives and precision with an explicit undefined case. Count predicted positives against actual negatives.

Hint 1: Use tp + fp as precision’s denominator.
Hint 2: Leave recall and its absent-class behavior unchanged.

Solution, tests and trade-offs

Fixtures cover missed positives, a correct positive with an extra false alarm and an all-negative batch.

Fixtures cover missed positives, a correct positive with an extra false alarm and an all-negative batch. Practice inspects false positives and precision as well.

These asymmetric cases distinguish the two denominators. Metrics are computed on supplied labels only; they do not establish label quality, fair subgroup coverage, calibration or the business cost of each error.

Practice solution and acceptance checks

python

from bootstrap_lab import torch_setup
torch = torch_setup()

def binary_metrics(predictions, targets):
    if len(predictions) != len(targets) or not targets:
        raise ValueError('matching nonempty batches required')
    if any(type(v) is not int or v not in (0, 1) for v in predictions + targets):
        raise ValueError('binary integer classes required')
    predicted = torch.tensor(predictions)
    actual = torch.tensor(targets)
    tp = ((predicted == 1) & (actual == 1)).sum().item()
    fn = ((predicted == 0) & (actual == 1)).sum().item()
    fp = ((predicted == 1) & (actual == 0)).sum().item()
    return {'fp': fp, 'precision': tp / (tp + fp) if tp + fp else None, 'accuracy': (predicted == actual).float().mean().item(),
            'recall': tp / (tp + fn) if tp + fn else None,
            'tp': tp, 'fn': fn}

Reasoning: Fixtures cover missed positives, a correct positive with an extra false alarm and an all-negative batch. Practice inspects false positives and precision as well.

Interview reasoning and next step

Choose metrics according to the decision being supported, report class support and keep threshold tuning separate from final held-out evaluation..

Choose metrics according to the decision being supported, report class support and keep threshold tuning separate from final held-out evaluation.

For multiclass problems consider per-class and macro summaries rather than only one aggregate number. In a job recommendation setting, missing a suitable role and surfacing an unsuitable one may have different costs, so explain that tradeoff alongside numerical scores.

Optional stretch — after the core task works: Compare macro and support-weighted summaries on a small multiclass fixture.

Computer practice: https://applydjinn.com/typorithm/python-frameworks-lab-20261105.zip. Install the pinned dependencies in an isolated environment, run verify.py --track torch, then edit the retained exercise baseline and check it with --practice pt-metrics. The lab uses owned in-memory databases, local framework clients and tiny CPU tensors. It never reads the working ApplyDjinn database, contacts production feeds or downloads models.

Inspect class-specific failures instead of trusting accuracy alone.