PyTorch: autograd and deliberate gradient accumulation

pythontorchindependent-practice

Saved on this device

Full lesson · 4 learning steps Open any exercise for focused practice

Step 1 of 4 · intro card

PyTorch: autograd and deliberate gradient accumulation

Concept and contract

Build a fresh graph and reset gradients when an update should be independent.

Autograd records differentiable operations involving tensors that require gradients and propagates derivatives backward through the resulting graph.

A leaf tensor’s grad field accumulates contributions by default. That accumulation can be useful for deliberate microbatching, but it is a bug when each independent update is supposed to start with a fresh gradient buffer.

Mechanism and worked example

The helper differentiates the scalar square function and optionally repeats an independent computation.

The helper differentiates the scalar square function and optionally repeats an independent computation. Before the second backward call it clears the leaf gradient, and it constructs a fresh square expression rather than reusing a freed graph.

The expected derivative is twice the input value. This simple analytic target makes accumulation errors visible without a neural network or optimizer.

Complete worked example

python

from bootstrap_lab import torch_setup
torch = torch_setup()

def gradient_at(value, repeat=False):
    x = torch.tensor(float(value), requires_grad=True)
    (x ** 2).backward()
    if repeat:
        x.grad = None
        (x ** 2).backward()
    return x.grad.item()

Reasoning: The helper differentiates the scalar square function and optionally repeats an independent computation. Before the second backward call it clears the leaf gradient, and it constructs a fresh square expression rather than reusing a freed graph.

Counterexamples and common mistakes

The counterexample removes the reset, so a repeated call doubles the gradient..

The counterexample removes the reset, so a repeated call doubles the gradient.

Confusing graph lifetime with gradient-buffer lifetime is common: retain_graph changes saved graph behavior, not the intended accumulation policy. Converting an intermediate to a Python number with item also detaches that numerical value from future graph operations, so use it for observation rather than inside a differentiable loss path.

Counterexample: a runnable framework mistake

python

from bootstrap_lab import torch_setup
torch = torch_setup()

def gradient_at(value, repeat=False):
    x = torch.tensor(float(value), requires_grad=True)
    (x ** 2).backward()
    if repeat:
        pass  # accumulated gradient retained
        (x ** 2).backward()
    return x.grad.item()

Reasoning: The counterexample removes the reset, so a repeated call doubles the gradient.

Independent practice: one bounded framework change

Change the differentiated function from x squared to x cubed in both backward expressions.

Change the differentiated function from x squared to x cubed in both backward expressions. The expected derivative becomes three times x squared, and a repeated independent computation should still report one derivative contribution.

Preserve the reset and fresh graph. This small analytic change reinforces the distinction between function shape, graph construction and gradient-buffer state.

Step 1: Differentiate x cubed while preserving independent gradient resets. Step 2: Change both fresh expressions.

Practice starter: solve before reading the solution

python

from bootstrap_lab import torch_setup
torch = torch_setup()

def gradient_at(value, repeat=False):
    # Remove this placeholder, then adapt the retained baseline:
    # Differentiate x cubed while preserving independent gradient resets.
    raise NotImplementedError
    x = torch.tensor(float(value), requires_grad=True)
    (x ** 2).backward()
    if repeat:
        x.grad = None
        (x ** 2).backward()
    return x.grad.item()

Reasoning: Differentiate x cubed while preserving independent gradient resets. Change both fresh expressions.

Hint 1: Use the analytic derivative 3*x*x.
Hint 2: Keep the repeated mode from accumulating twice.

Solution, tests and trade-offs

Fixtures cover positive and negative inputs and the repeated-gradient mode.

Fixtures cover positive and negative inputs and the repeated-gradient mode. Practice checks cubic derivatives and a zero input.

Compare numerical gradients with the analytic values using finite tolerance. These checks do not substitute for numerical gradient checking on a custom complex operation, but they establish the basic accumulation and reset behavior on the pinned CPU implementation.

Practice solution and acceptance checks

python

from bootstrap_lab import torch_setup
torch = torch_setup()

def gradient_at(value, repeat=False):
    x = torch.tensor(float(value), requires_grad=True)
    (x ** 3).backward()
    if repeat:
        x.grad = None
        (x ** 3).backward()
    return x.grad.item()

Reasoning: Fixtures cover positive and negative inputs and the repeated-gradient mode. Practice checks cubic derivatives and a zero input.

Interview reasoning and next step

In a training loop, clear optimizer gradients before the intended accumulation window, compute a scalar loss, call backward and then step.

In a training loop, clear optimizer gradients before the intended accumulation window, compute a scalar loss, call backward and then step. If you intentionally accumulate across microbatches, define loss scaling and update frequency explicitly.

Keep inference paths under suitable gradient-disabled contexts. The next module and optimizer lessons connect this small derivative model to registered parameters and actual updates.

Optional stretch — after the core task works: Compare an intentional two-microbatch accumulation with a single combined batch.

Computer practice: https://applydjinn.com/typorithm/python-frameworks-lab-20261105.zip. Install the pinned dependencies in an isolated environment, run verify.py --track torch, then edit the retained exercise baseline and check it with --practice pt-autograd. The lab uses owned in-memory databases, local framework clients and tiny CPU tensors. It never reads the working ApplyDjinn database, contacts production feeds or downloads models.

Build a fresh graph and reset gradients when an update should be independent.