How computer vision learned to go deep
The road to ResNet: what the field tried for fifty years, the wall it hit at twenty layers, and the one-line change that removed it.
Six linked notes on how computer vision got from hand-drawn edge detectors to 152-layer networks, why depth stopped working at around twenty layers, and how one change to what a layer is asked to compute removed the ceiling. Written from the primary sources, with every figure traceable to a published number.
Loading figure. It needs JavaScript; the surrounding text stands on its own without it.
The two threads ran in parallel for decades. Classical vision had the benchmarks and the credibility; the neural lineage had the right architecture and no way to train it. They met when ImageNet supplied the data and GPUs supplied the compute.
Read in order
Numbered by the order the notes were written, which is not the order to read them. Start with part 1 for the chronological path, or jump straight to part 4 if you only want the paper.
- Part 1 The classical era of computer vision From the blocks world to Fisher vectors: fifty years of hand-designed representations, and the ceiling they hit.
- Part 2 The neural lineage and the 2012 break A thread running since 1959 that spent most of its life out of favour, then displaced everything in eighteen months.
- Part 3 The challenge facing researchers in 2015 Depth was the best known lever on accuracy, and nobody could use more than about twenty layers of it. The blocker was not the one everybody remembers.
- Part 4 Deep residual learning He, Zhang, Ren and Sun, December 2015. Stop asking layers to learn the identity, and depth stops being a constraint.
- Part 5 What it costs to train ResNet today A result that took a research team weeks of cluster time in 2015 is now a same-day errand for about twenty dollars. Plus: how slow is a Mac, really?
- Part 6 ResNet from scratch, and the experiment worth running Working PyTorch for the blocks, the full architectures, and a reproduction of the degradation result that motivated the whole paper.
The thread in one paragraph
Classical vision hand-designed its representations and could only learn the final classifier, which capped progress no matter how good the engineering got . A neural lineage running since 1959 had the right architecture for learning representations but lacked the data, compute, and training methods until roughly 2011, at which point AlexNet ended the argument and each subsequent ILSVRC winner was deeper than the last . That made depth the field's central variable, but deeper plain networks mysteriously produced higher training error, an optimization failure rather than overfitting, and every available workaround added training machinery without solving it . ResNet resolved it by changing what each block is asked to learn: wire the identity through as a parameter-free shortcut and learn only the residual on top .
The challenge ResNet solved was not vanishing gradients. That problem was already handled by 2015 through normalized initialization and batch normalization, and the paper says so explicitly. The real blocker was degradation: deeper plain networks reached higher training error than shallower ones, which rules out overfitting, capacity and data as explanations and leaves optimization difficulty. Since a deeper network can always match a shallower one by setting its extra layers to the identity, a good solution provably exists and SGD simply could not find it. Residual blocks make that solution the default instead of something to be discovered.
What each part contributes
| Part | Question it answers | Interactive figure |
|---|---|---|
| 1. Classical era | What were people doing before deep learning, and why did it stall? | Pipeline diagram: one learned stage out of four |
| 2. Neural lineage | Where did convolutional networks come from, and why did they win in 2012 and not 1998? | ILSVRC error and depth by year |
| 3. The challenge | What exactly was blocking deeper networks in 2015? | Gradient flow simulation, degradation in three views, the identity construction |
| 4. ResNet | What did the paper actually change, and what did it buy? | Plain vs residual block, accuracy against cost |
| 5. What it costs now | What would it take to reproduce this today, including on a Mac? | Training budget calculator |
| 6. Code | How do I build it and reproduce the key result myself? | Working PyTorch, and the experiment worth running |
Conventions and sourcing
- Figures labelled with paper table numbers use exact published values. Figures labelled illustrative are simulations that convey a mechanism and are not reproductions of data.
- Compute is quoted in multiply-adds, the convention used in the ResNet paper, with the equivalent in raw floating point operations given alongside where it matters.
- Where sources disagree, both numbers appear. The 2013 ILSVRC result is a good example: Clarifai's winning entry was about 11.2%, while the ZFNet paper's single model was 11.7%.
- These pages are generated from 7 markdown notes totalling about 11,758 words. Nothing in the source notes is omitted here.