PyTorch capstone: a bounded classifier with held-out evaluation

pythontorchindependent-practice

Saved on this device

Full lesson · 4 learning steps Open any exercise for focused practice

Step 1 of 4 · intro card

PyTorch capstone: a bounded classifier with held-out evaluation

Concept and contract

Connect a complete tiny training loop to a separate observed prediction set.

A complete learning experiment needs more than a model definition. It should identify its training population, objective, update budget and separate evaluation population, then report evidence from the actual trained model.

This capstone uses a deliberately simple sign-classification task so the whole flow remains inspectable on CPU. It is a learning exercise, not a production hiring or candidate-scoring system.

Mechanism and worked example

The fixture trains a one-input two-logit linear classifier on four owned points with known labels.

The fixture trains a one-input two-logit linear classifier on four owned points with known labels. Parameters start at zero, SGD updates are bounded and cross-entropy receives raw logits.

Two different points are held out of every update. After training, eval and inference_mode produce predictions, while the helper reports whether training loss fell and whether evaluation mode was actually set.

Complete worked example

python

from bootstrap_lab import torch_setup
torch = torch_setup()

from torch import nn

def tiny_classifier(steps=60):
    if type(steps) is not int or not 1 <= steps <= 100:
        raise ValueError('bounded training steps required')
    train_x = torch.tensor([[-2.],[-1.],[1.],[2.]])
    train_y = torch.tensor([0,0,1,1], dtype=torch.long)
    held_x = torch.tensor([[-1.5],[1.5]])
    held_y = torch.tensor([0,1], dtype=torch.long)
    model = nn.Linear(1, 2)
    with torch.no_grad():
        model.weight.zero_()
        model.bias.zero_()
    optimizer = torch.optim.SGD(model.parameters(), lr=0.2)
    loss_fn = nn.CrossEntropyLoss()
    initial = loss_fn(model(train_x), train_y).item()
    model.train()
    for _ in range(steps):
        optimizer.zero_grad(set_to_none=True)
        loss = loss_fn(model(train_x), train_y)
        loss.backward()
        optimizer.step()
    model.eval()
    with torch.inference_mode():
        final = loss_fn(model(train_x), train_y).item()
        predictions = model(held_x).argmax(dim=1)
    return {'loss_fell': final < initial, 'predictions': predictions.tolist(),
            'training': model.training}

Reasoning: The fixture trains a one-input two-logit linear classifier on four owned points with known labels. Parameters start at zero, SGD updates are bounded and cross-entropy receives raw logits.

Counterexamples and common mistakes

The counterexample removes optimizer.step, leaving the model at its initial symmetric output.

The counterexample removes optimizer.step, leaving the model at its initial symmetric output. A loss can be computed repeatedly without learning, and a loop counter alone proves nothing.

Likewise, training accuracy would not replace held-out evaluation. The tiny points are intentionally separable; perfect results here should not be generalized to noisy real data, fairness claims or a broad labor-market prediction product.

Counterexample: a runnable framework mistake

python

from bootstrap_lab import torch_setup
torch = torch_setup()

from torch import nn

def tiny_classifier(steps=60):
    if type(steps) is not int or not 1 <= steps <= 100:
        raise ValueError('bounded training steps required')
    train_x = torch.tensor([[-2.],[-1.],[1.],[2.]])
    train_y = torch.tensor([0,0,1,1], dtype=torch.long)
    held_x = torch.tensor([[-1.5],[1.5]])
    held_y = torch.tensor([0,1], dtype=torch.long)
    model = nn.Linear(1, 2)
    with torch.no_grad():
        model.weight.zero_()
        model.bias.zero_()
    optimizer = torch.optim.SGD(model.parameters(), lr=0.2)
    loss_fn = nn.CrossEntropyLoss()
    initial = loss_fn(model(train_x), train_y).item()
    model.train()
    for _ in range(steps):
        optimizer.zero_grad(set_to_none=True)
        loss = loss_fn(model(train_x), train_y)
        loss.backward()
        pass  # model never updated
    model.eval()
    with torch.inference_mode():
        final = loss_fn(model(train_x), train_y).item()
        predictions = model(held_x).argmax(dim=1)
    return {'loss_fell': final < initial, 'predictions': predictions.tolist(),
            'training': model.training}

Reasoning: The counterexample removes optimizer.step, leaving the model at its initial symmetric output. A loss can be computed repeatedly without learning, and a loop counter alone proves nothing.

Independent practice: one bounded framework change

Add held_out_accuracy using the already computed predictions and held_y labels.

Add held_out_accuracy using the already computed predictions and held_y labels. Keep held-out examples outside every optimization step and preserve the bounded step validation.

This small metric addition completes the visible evaluation contract. Do not train on held_y to make the score improve, and do not substitute the expected labels for the observed model predictions.

Step 1: Report held-out accuracy without using held-out labels during training. Step 2: Compare observed predictions with held_y after evaluation.

Practice starter: solve before reading the solution

python

from bootstrap_lab import torch_setup
torch = torch_setup()

from torch import nn

def tiny_classifier(steps=60):
    # Remove this placeholder, then adapt the retained baseline:
    # Report held-out accuracy without using held-out labels during training.
    raise NotImplementedError
    if type(steps) is not int or not 1 <= steps <= 100:
        raise ValueError('bounded training steps required')
    train_x = torch.tensor([[-2.],[-1.],[1.],[2.]])
    train_y = torch.tensor([0,0,1,1], dtype=torch.long)
    held_x = torch.tensor([[-1.5],[1.5]])
    held_y = torch.tensor([0,1], dtype=torch.long)
    model = nn.Linear(1, 2)
    with torch.no_grad():
        model.weight.zero_()
        model.bias.zero_()
    optimizer = torch.optim.SGD(model.parameters(), lr=0.2)
    loss_fn = nn.CrossEntropyLoss()
    initial = loss_fn(model(train_x), train_y).item()
    model.train()
    for _ in range(steps):
        optimizer.zero_grad(set_to_none=True)
        loss = loss_fn(model(train_x), train_y)
        loss.backward()
        optimizer.step()
    model.eval()
    with torch.inference_mode():
        final = loss_fn(model(train_x), train_y).item()
        predictions = model(held_x).argmax(dim=1)
    return {'loss_fell': final < initial, 'predictions': predictions.tolist(),
            'training': model.training}

Reasoning: Report held-out accuracy without using held-out labels during training. Compare observed predictions with held_y after evaluation.

Hint 1: Keep the training loop unchanged.
Hint 2: Retain the explicit step budget and negative case.

Solution, tests and trade-offs

Fixtures compare several bounded training budgets and require loss reduction, correct held-out predictions and training=False.

Fixtures compare several bounded training budgets and require loss reduction, correct held-out predictions and training=False. Practice also checks held_out_accuracy equals one on this synthetic task.

An invalid zero-step request must fail before training. These checks connect actual updates, observed loss and selected predictions; they do not certify real-world model quality or deployment performance.

Practice solution and acceptance checks

python

from bootstrap_lab import torch_setup
torch = torch_setup()

from torch import nn

def tiny_classifier(steps=60):
    if type(steps) is not int or not 1 <= steps <= 100:
        raise ValueError('bounded training steps required')
    train_x = torch.tensor([[-2.],[-1.],[1.],[2.]])
    train_y = torch.tensor([0,0,1,1], dtype=torch.long)
    held_x = torch.tensor([[-1.5],[1.5]])
    held_y = torch.tensor([0,1], dtype=torch.long)
    model = nn.Linear(1, 2)
    with torch.no_grad():
        model.weight.zero_()
        model.bias.zero_()
    optimizer = torch.optim.SGD(model.parameters(), lr=0.2)
    loss_fn = nn.CrossEntropyLoss()
    initial = loss_fn(model(train_x), train_y).item()
    model.train()
    for _ in range(steps):
        optimizer.zero_grad(set_to_none=True)
        loss = loss_fn(model(train_x), train_y)
        loss.backward()
        optimizer.step()
    model.eval()
    with torch.inference_mode():
        final = loss_fn(model(train_x), train_y).item()
        predictions = model(held_x).argmax(dim=1)
    accuracy = (predictions == held_y).float().mean().item()
    return {'held_out_accuracy': accuracy, 'loss_fell': final < initial, 'predictions': predictions.tolist(),
            'training': model.training}

Reasoning: Fixtures compare several bounded training budgets and require loss reduction, correct held-out predictions and training=False. Practice also checks held_out_accuracy equals one on this synthetic task.

Interview reasoning and next step

To continue, introduce a noisier owned dataset, a time- or group-aware split, saved preprocessing, class-specific metrics and checkpoint restoration.

To continue, introduce a noisier owned dataset, a time- or group-aware split, saved preprocessing, class-specific metrics and checkpoint restoration. Keep resource budgets explicit and compare against a simple baseline before increasing complexity.

Explain limitations alongside results. The course’s storage, gradient, registration, loss, data, evaluation and export lessons provide separate regression checks for the pieces combined in this experiment.

Optional stretch — after the core task works: Add a copied-state checkpoint round trip and verify the same held-out predictions.

Computer practice: https://applydjinn.com/typorithm/python-frameworks-lab-20261105.zip. Install the pinned dependencies in an isolated environment, run verify.py --track torch, then edit the retained exercise baseline and check it with --practice pt-capstone. The lab uses owned in-memory databases, local framework clients and tiny CPU tensors. It never reads the working ApplyDjinn database, contacts production feeds or downloads models.

Connect a complete tiny training loop to a separate observed prediction set.