I Built the Step I Needed Before micrograd

Andrej Karpathy's micrograd lecture builds a small neural network and the machinery that lets it learn. It assumes basic Python and some remembered high-school calculus.

My Python was comfortable. My derivatives were rusty. Following the mathematics and the code at the same time meant holding too many new pieces at once. I kept losing the thread.

So I built ml-from-scratch. It has four Jupyter notebooks that use synthetic basketball data to predict a team's points from offensive rating, pace, and an opponent's defensive rating. The fourth notebook adapts Karpathy's micrograd engine with its attribution and license notices retained.

I still wanted the lecture. I needed one smaller step before it.

I started with one adjustable number in a training loop, then expanded to four. The learning rule moves that number in the direction that reduces prediction error. I use a derivative to see how the error changes when the number moves. The step size controls how far we move; too large a step can overshoot.

Next I checked a calculated derivative by nudging an input, measuring the change in output, and dividing by the size of the nudge. That numerical result catches mistakes I can miss in the algebra.

The third notebook adds a hidden layer and raises the adjustable values from four to sixteen. It's still small enough to inspect.

In the last notebook, I build automatic differentiation. The code records each operation and then works backward to compute derivatives.

Take a value, multiply it by three, and then double the result. Increasing the starting value by one raises the final result by six. The two local effects combine: three times two. If the same value reaches the result through another path, that contribution has to be added too.

The code keeps those connections and uses them to carry each effect backward through a larger calculation. That's the work behind “backpropagation” in this small system.

If you can read Python but the calculus is rusty, this is the step I wanted. If Python itself is new, start somewhere else.

Explain one update in the first notebook. Say which number moved, why that direction should reduce the error, and what the step size controlled.

If the answer is still copied arithmetic, change one input and follow the result. I moved on when I could explain why the number changed instead of just repeating the calculation.

Sources: Karpathy's course prerequisites; the four-notebook README.