Part 2 · 1959 to early 2015

The neural lineage and the 2012 break

A thread running since 1959 that spent most of its life out of favour, then displaced everything in eighteen months.

Source: notes/03-cv-history-deep-learning-lineage.md · 1,860 words · 1959 to early 2015

The classical pipeline covered in Classical era was not the only thread running. A neural network lineage had existed since the late 1950s, spent most of that time out of favor, and then displaced everything in about eighteen months. This note traces that thread up to the eve of ResNet.

#1959 to 1968: the biology that started it

Hubel and Wiesel recorded from single neurons in cat visual cortex and found a hierarchy. "Simple cells" responded to edges at a specific orientation in a specific location. "Complex cells" responded to the same orientation but tolerated shifts in position. Cells further up the hierarchy responded to progressively more complex patterns over progressively larger receptive fields, with progressively more invariance to position. They received a Nobel Prize in 1981.

Three architectural ideas in every modern vision network come directly from this work: local receptive fields, a hierarchy from simple to complex features, and alternating detection and pooling stages that trade spatial precision for invariance.

#1958 to 1979: perceptrons, the first winter, and the Neocognitron

Rosenblatt's perceptron (1958) was a single-layer learner and generated a first wave of public excitement. Minsky and Papert's Perceptrons (1969) documented what single-layer models cannot do, XOR being the canonical example, and the resulting loss of confidence contributed to the first AI winter. The multilayer case was the answer, but no one had a practical way to train multilayer networks yet.

Fukushima's Neocognitron (1979, published 1980) is the direct ancestor of the CNN. It implemented Hubel and Wiesel's hierarchy explicitly: "S-cells" performing local convolution-style feature detection with replicated weights, alternating with "C-cells" performing spatial averaging for shift tolerance, stacked into a genuinely deep network. Weight replication meant few parameters described a whole layer.

What it lacked was backpropagation. Fukushima trained it with local, winner-take-all unsupervised rules, because supervised training of deep networks was not available in 1979. Architecturally it is remarkably close to a modern convnet; the missing piece was purely the learning algorithm.

#1986 to 1998: backpropagation and LeNet

Reverse-mode automatic differentiation was published by Linnainmaa in 1970, and several researchers applied related ideas to networks through the early 1980s. Rumelhart, Hinton, and Williams' 1986 paper is what popularized backpropagation for multilayer networks and made supervised deep training a practical option.

The convolutional application followed quickly:

• Waibel's time-delay neural networks (1987) used 1D convolutions with weight sharing for speech, trained by backpropagation.
• Zhang et al. (1988) built what is generally recognized as the first modern 2D CNN trained by backpropagation, applied to character recognition.
• LeCun et al. (1989) at Bell Labs trained a CNN by backpropagation to read handwritten ZIP codes.
• Max pooling replaced Fukushima's spatial averaging in the early 1990s (Yamaguchi et al. for 1D, Weng et al. for 2D).

LeNet-5 (LeCun, Bottou, Bengio, Haffner, 1998) is the canonical endpoint of this phase: an end-to-end trainable document recognition system reaching about 0.95% test error on MNIST, with roughly 345,000 connections but only about 60,000 free parameters thanks to weight sharing. It was deployed commercially to read bank checks. The 1998 paper states the thesis plainly: because all weights are learned by backpropagation, convolutional networks "can be seen as synthesizing their own feature extractor."

That sentence is the whole argument against the classical pipeline, made fourteen years before the field accepted it.

#Late 1990s to 2006: the second winter for neural vision

LeNet worked on 32x32 grayscale digits and did not transfer to natural images at the time. Meanwhile SVMs and kernel methods offered convex optimization, generalization theory, and reproducible results. Against that, neural networks looked unattractive: non-convex training with no guarantees, sensitivity to initialization and learning rates, a reputation for requiring impractical amounts of data and compute, and no theory to justify the extra complexity. Papers were hard to publish and the term "neural network" itself became a liability.

Two things kept the thread alive. First, layerwise unsupervised pretraining with deep belief nets (Hinton, Osindero, Teh, 2006) demonstrated that deep networks could be trained at all if you were clever about initialization, and popularized the rebranding to "deep learning." Second, and more decisively, GPUs arrived.

#2009 to 2011: GPUs prove depth works, before ImageNet

This period is often skipped, but it matters: the evidence that deep supervised CNNs work predates AlexNet. Cireşan, Meier, Masci, Gambardella, and Schmidhuber at IDSIA built a fast, fully parameterizable CNN implementation on NVIDIA GPUs and reported:

• MNIST at 0.35% error with a plain deep CNN trained by ordinary backpropagation (IJCAI 2011), improving to 0.31% with a committee and 0.23% with a multi-column network in 2012.
• Best published results on NORB (2.53%) and CIFAR-10 (19.51%).
• The German Traffic Sign Recognition Benchmark in 2011 at 99.46% accuracy, against a measured human rate of 98.84%. This is generally counted as the first better-than-human result in a visual recognition competition. Their multi-column model used 25 columns trained on differently preprocessed inputs and took 37 hours on four GPUs.

Their IJCAI paper is blunt about the conclusion: "Deep nets trained by simple back-propagation perform better than more shallow ones," and "This shows once more that deep networks, contrary to common belief, can be trained successfully by backpropagation." The GPU implementation is what made it testable, reducing training from months to days.

So by 2011 the ingredients were all present and publicly demonstrated. What was missing was a result on a benchmark the mainstream vision community cared about.

#2012: AlexNet

Krizhevsky, Sutskever, and Hinton entered ILSVRC 2012 with an 8-layer CNN: five convolutional layers, three fully connected, about 60 million parameters. The result was 15.3% top-5 error against roughly 26.2% for the runner-up, a ten-point absolute gap of a kind that benchmark had never seen.

The ingredients were mostly not novel individually; the combination and the execution were:

• ReLU activations instead of tanh or sigmoid, which trained several times faster and did not saturate.
• Dropout in the fully connected layers, which made a 60M-parameter model trainable on 1.2M images without collapsing into overfitting.
• GPU training on two GTX 580 cards with 3GB each, for roughly a week. The famous two-column architecture diagram exists because the model did not fit on one card, so the channels were split across GPUs.
• Aggressive data augmentation: random crops, horizontal flips, and color jitter along PCA directions of the RGB distribution.
• Overlapping max pooling and local response normalization.

The significance was not the architecture. It was that a learned representation, trained end-to-end from pixels, beat the best hand-engineered pipeline in the world by a margin that could not be argued with. The classical pipeline did not decline gradually; ILSVRC 2013 was almost entirely CNN submissions.

#2013: ZFNet and the beginning of understanding

Zeiler and Fergus won ILSVRC 2013 as Clarifai at roughly 11.2% top-5 error. Their contribution was diagnostic rather than architectural: a deconvolutional visualization technique that projected activations back to pixel space to show what individual filters responded to. Using it, they found AlexNet's first layer had dead and aliased filters caused by an overly aggressive 11x11 stride-4 configuration, and fixed it with 7x7 stride 2. This marked the start of treating architecture design as something you could debug with evidence instead of guess at.

The same period established that CNN features transfer. R-CNN (Girshick et al., 2014) used region proposals plus AlexNet-family features for detection and roughly doubled PASCAL VOC 2007 detection mAP relative to DPM, moving from the low 30s to the high 50s. DeCAF, OverFeat, and SPPnet made the same point in other tasks. The practical consequence was that classification backbone quality became the bottleneck for every vision task, which is precisely why the next two years of research fixated on making backbones better.

#2014: depth becomes the explicit variable

Two ILSVRC 2014 entries defined the question ResNet would answer.

VGG (Simonyan and Zisserman) made depth the headline. Its design was radically uniform: only 3x3 convolutions, stacked, with channel counts doubling as spatial resolution halved, at 16 or 19 weight layers, reaching about 7.3% top-5 error. The insight was that a stack of small filters has the same receptive field as one large filter but with more nonlinearities and fewer parameters. The cost was enormous: VGG-16 carries about 138 million parameters and roughly 15 to 20 GFLOPs.

One detail from the VGG paper matters a great deal for the ResNet story. They could not train the 16- and 19-layer configurations from scratch reliably. They first trained a shallower 11-layer network, then used its weights to initialize the deeper ones, adding layers in stages. Depth was already visibly at the edge of trainability at 19 layers.

GoogLeNet / Inception v1 (Szegedy et al.) won at 6.66% top-5 with 22 layers and only about 5 million parameters, roughly a twelfth of VGG's, via inception modules that ran parallel filter sizes and used 1x1 convolutions as channel bottlenecks, plus global average pooling instead of large fully connected layers.

The other detail that matters: GoogLeNet attached auxiliary classifiers to intermediate layers during training, with their own losses, for the explicit purpose of injecting gradient into the middle of the network. Like VGG's staged initialization, this is a workaround for the fact that gradients were not reaching early layers well at 22 layers of depth.

#Early 2015: the last pieces before ResNet

• Batch Normalization (Ioffe and Szegedy, February 2015) normalized layer inputs per mini-batch, which stabilized training, allowed much higher learning rates, and reduced the need for careful initialization. BN-Inception reached about 4.9% top-5.
• He initialization (He et al., February 2015, in the PReLU paper) derived the correct initialization scale for ReLU networks specifically, where the earlier Xavier/Glorot derivation assumed symmetric activations. That paper reported about 4.94% top-5, the first claim of surpassing the frequently cited ~5.1% human benchmark on this task.

With BN and proper initialization, very deep plain networks would finally converge. The surprise waiting on the other side of that is the subject of The challenge.

#The ILSVRC scoreboard, condensed

The pattern across 2012 to 2014 is unambiguous: every winner was deeper than the last, and depth was the variable most reliably correlated with accuracy. That is what set up the question at the top of the ResNet paper: is learning better networks as easy as stacking more layers?