← All lessons

Lesson 3 — Neural Networks by Hand

Where Lesson 2 ends: TD(0) learns V from played games instead of a model — but it still keeps one number per state in a table. Asteroid Hop has 11 entries; that's fine. Star Realms has on the order of 1030 states, and — the sharper problem — you never visit the same state twice, so a table can be neither stored nor filled. Every lookup is a miss. This lesson removes the table.

The idea: replace the table with a function

Instead of storing V(s) in a cell, compute it from features of s with a parameterized function Vθ(s) — a neural network. The win is generalization: tune θ on the states your games do visit, and the function returns sensible values for the overwhelming majority you'll never see, because nearby states share features. The TD update barely changes — it stops being "nudge a cell" and becomes "nudge the weights so the output moves toward the TD target," via the gradient.

θ ← θ + α·[ r + γ·Vθ(s′) − Vθ(s) ]·∇θ Vθ(s)

The bracket is the same TD error δ from Lesson 2; the new factor is the gradient, which spreads each correction across every state the network judges similar. To trust it, you'll build the network — forward pass, loss, and backpropagation — by hand, in NumPy, before ever importing a framework.

The exercise — coming with the live session

The full write-up and stub land when this session runs. The shape of the assignment:

  1. A tiny network from scratch. One hidden layer, implemented as plain matrix multiplies — forward pass, a squared-error loss against the TD target, and the backward pass computed by hand.
  2. Gradient-check it. Compare your analytic gradients to numerical finite differences, so you know the backprop is right before trusting it.
  3. Re-learn Asteroid Hop's V with the network instead of the 11-entry table, and confirm it reproduces the Lesson 1 values — same toy, known ground truth, now via function approximation.
  4. Predictions first, as always — including what the deadly triad (bootstrapping + function approximation + off-policy data) might do to convergence.

← All lessons