Part 3 · 2014 to 2015
The challenge facing researchers in 2015
Depth was the best known lever on accuracy, and nobody could use more than about twenty layers of it. The blocker was not the one everybody remembers.
By mid-2015 the field had a clear belief and a hard wall. The belief, earned over three years of ILSVRC results, was that depth was the most reliable route to accuracy. The wall was that nobody could actually train networks much deeper than about 20 to 30 layers and get a benefit. This note lays out what specifically was blocking them, which is the setup for ResNet.
The question, stated as the ResNet paper states it
Driven by the significance of depth, a question arises: Is learning better networks as easy as stacking more layers?
Everything below is the list of reasons the honest answer in mid-2015 was "no."
Obstacle 1: vanishing and exploding gradients (mostly solved)
Loading figure. It needs JavaScript; the surrounding text stands on its own without it.
Illustrative simulation, not measured data. A plain network multiplies a fresh Jacobian at each layer, so the backward signal drifts geometrically once the per-layer gain leaves 1. A residual network multiplies terms centred on 1, because the identity path contributes a fixed 1 to every derivative. Toggle the regime and watch what happens to the argument.
Backpropagation multiplies Jacobians layer by layer, so gradient magnitude tends to decay or blow up geometrically with depth. If it decays, early layers receive almost no learning signal and the network never converges from the start. This was the well-known problem, traced back to Bengio et al. (1994) and Hochreiter's work on recurrent networks.
By the time of ResNet this was largely handled, and the paper says so explicitly, crediting two families of fixes:
- Normalized initialization: LeCun (1998), Xavier/Glorot (2010), orthogonal initialization (Saxe et al., 2013), and He initialization (2015), the last of which corrected the variance scaling for ReLU networks specifically.
- Intermediate normalization layers: Batch Normalization (Ioffe and Szegedy, 2015), applied right after each convolution and before the activation.
Together these "enable networks with tens of layers to start converging for SGD with backpropagation." Note the careful phrasing: start converging, and tens of layers. Both qualifiers were doing real work.
It is also worth noting how much scaffolding the 2014 architectures needed for gradient flow, which shows the problem was controlled rather than eliminated:
- VGG-16 and VGG-19 could not be trained from scratch reliably. The authors trained an 11-layer network first and used it to initialize the deeper configurations in stages.
- GoogLeNet attached auxiliary classifiers with their own losses to intermediate layers, purely to inject gradient into the middle of a 22-layer network.
Neither is a principled solution. Both are workarounds bolted onto the training procedure.
Obstacle 2: the degradation problem (unsolved, and the real target)
Loading figure. It needs JavaScript; the surrounding text stands on its own without it.
The first view is the cleanest evidence in the paper and uses exact published numbers. Sixteen extra layers made the plain network worse, and the same sixteen layers made the residual network substantially better. Both models saw identical data and identical training.
Once BN and good initialization let deep plain networks converge, a stranger failure appeared. In the paper's words:
When deeper networks are able to start converging, a degradation problem has been exposed: with the network depth increasing, accuracy gets saturated (which might be unsurprising) and then degrades rapidly. Unexpectedly, such degradation is not caused by overfitting, and adding more layers to a suitably deep model leads to higher training error.
The diagnostic detail is the one to remember. Training error went up with depth.
That single fact rules out the explanations a practitioner would reach for first:
| Candidate explanation | Why it is ruled out |
|---|---|
| Overfitting | Overfitting means low training error and high test error. Here training error itself rose. |
| Vanishing gradients | These networks had BN and proper initialization, and were converging. The paper explicitly reports that degradation was not caused by vanishing gradients. |
| Insufficient capacity | The deeper network strictly contains the shallower one as a special case, so it has more capacity, not less. |
| Insufficient data | Same data in both runs, and the failure shows up on the training set. |
Concrete numbers from the paper's experiments: on CIFAR-10, a 56-layer plain CNN had higher training and test error than an otherwise identical 20-layer one. On ImageNet, a 34-layer plain network performed worse than an 18-layer plain network.
The identity-mapping argument that made it a paradox
Loading figure. It needs JavaScript; the surrounding text stands on its own without it.
Because the extra layers could simply compute the identity, a solution at least as good as the shallow network provably exists in the deeper network's parameter space. When SGD returns something worse, the failure cannot be about capacity, data, or generalization. It is the optimizer failing to find a solution that is known to be there.
This is the cleanest piece of reasoning in the paper, and it converts a disappointing empirical result into a precise statement about optimization:
Take a trained shallow network. Construct a deeper one by copying those layers and appending extra layers that compute the identity function. By construction, this deeper network computes exactly the same function and therefore achieves exactly the same training error. So a solution at least that good provably exists in the deeper network's parameter space.
Therefore, when SGD returns something worse, the failure cannot be about representational capacity, data, or generalization. It is that the optimizer cannot find a solution that is known to exist. The paper's conclusion: "The degradation of training accuracy indicates that not all systems are similarly easy to optimize."
Put differently: asking a stack of nonlinear layers with ReLUs and BN to learn an identity mapping turns out to be surprisingly hard, and deeper networks need many of their blocks to behave close to identity.
Obstacle 3: what the community tried instead
Several concurrent efforts attacked depth, and the shape of their solutions is informative: almost all of them added machinery to the training procedure rather than changing what a layer is asked to compute.
Highway Networks (Srivastava, Greff, Schmidhuber, May and July 2015) came closest. Taking inspiration from LSTM's gating, they added learned gates so information could flow along "information highways":
where T(x) is a learned transform gate. They successfully trained plain-versus-highway
comparisons at 10, 20, 50, and 100 layers and showed highway networks did not suffer from
increasing depth in the way plain networks did.
ResNet's own related-work section is precise about the difference, and it is the key conceptual distinction:
These gates are data-dependent and have parameters, in contrast to our identity shortcuts that are parameter-free. When a gated shortcut is "closed" (approaching zero), the layers in highway networks represent non-residual functions. On the contrary, our formulation always learns residual functions; our identity shortcuts are never closed, and all information is always passed through. In addition, highway networks have not demonstrated accuracy gains with extremely increased depth (e.g., over 100 layers).
So the gate was both the borrowed idea and the limiting one. A gate that can close can cut off the shortcut, and it costs parameters and compute, and in practice it did not deliver gains at extreme depth.
Other contemporaneous approaches:
- Deeply supervised nets (Lee et al., 2015): companion loss functions at intermediate layers, the same idea as GoogLeNet's auxiliary classifiers, generalized.
- FitNets (Romero et al., 2015): two-stage hint-based training to get thin deep networks to train at all.
- Staged and curriculum-style initialization: VGG's approach, growing a network by adding layers to an already-trained shallower one.
- Better initialization schemes such as layer-sequential unit-variance init, which improve the starting point but do not change the optimization landscape.
Older precedents in the same family, which the paper cites: adding a linear input-to-output layer in MLP training, and methods for centering layer responses and gradients.
Obstacle 4: the practical costs
Depth was also just expensive, which shaped what people were willing to try:
- VGG-19 needed roughly 19.6 GFLOPs per image, and VGG-16 carried about 138 million parameters, most of them in the fully connected layers. Memory on contemporary GPUs (3 to 12 GB) was a live constraint.
- Competitive ILSVRC entries were ensembles with heavy test-time augmentation, multi-crop and multi-scale evaluation, so each extra point of accuracy cost multiples of compute.
- With returns diminishing and then reversing past ~20 layers, there was no reason to pay for depth. The economics and the optimization failure reinforced each other.
Obstacle 5: the goalposts were moving under everyone
Two framing pressures made this urgent rather than academic:
- The ~5.1% top-5 human benchmark for ILSVRC classification (Russakovsky et al.) had already been passed in early 2015 by BN-Inception and PReLU-net at about 4.9%. The headline task was close to saturating, so the interesting question was shifting from "can we classify ImageNet" to "can we build a backbone good enough to carry detection, localization, and segmentation."
- Since R-CNN, every downstream vision task inherited its accuracy from the classification backbone. A better backbone improved detection, localization, and segmentation for free. This is why a classification architecture paper could win five competition tracks at once, and why backbone quality was the highest-leverage problem in the field.
Summary: the exact shape of the gap
The state of play in late 2015 was:
- Depth is empirically the best lever on accuracy, proven by three consecutive ILSVRC winners.
- Gradient flow at depth is mostly solved, by normalized initialization plus batch normalization.
- Despite that, deeper plain networks get worse training error, so the barrier is optimization difficulty, not capacity, data, or generalization.
- A provably-as-good solution exists in the deeper network's parameter space, built from identity mappings, and SGD cannot find it.
- Every available workaround added training machinery: gates, auxiliary losses, staged initialization, hint layers. None changed what a block was asked to learn, and none showed gains past about 100 layers.
Stated that way, the remaining move is almost forced. If the hard thing is learning an identity mapping, stop asking layers to learn it. Wire the identity in as a parameter-free shortcut and let the layers learn only the residual on top of it. That is ResNet.
Sources
- He, Zhang, Ren, Sun, "Deep Residual Learning for Image Recognition," arXiv:1512.03385.
- Srivastava, Greff, Schmidhuber, "Training Very Deep Networks," arXiv:1507.06228 and NIPS
- Russakovsky et al., "ImageNet Large Scale Visual Recognition Challenge," arXiv:1409.0575.
- Ioffe and Szegedy, "Batch Normalization," arXiv:1502.03167.
- He et al., "Delving Deep into Rectifiers" (PReLU and He initialization), arXiv:1502.01852.