Part 4 · 2015 to 2016

Deep residual learning

He, Zhang, Ren and Sun, December 2015. Stop asking layers to learn the identity, and depth stops being a constraint.

Source: notes/01-resnet-deep-residual-learning.md · 1,420 words · 2015 to 2016

Paper: Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun (Microsoft Research Asia). arXiv:1512.03385, December 2015. CVPR 2016 Best Paper.

Related notes: Classical era and Neural lineage cover how the field arrived here, and The challenge covers the degradation problem in more depth than the summary below.

#Background: how vision got deep

Before 2012, image classification was mostly hand-engineered features (SIFT, HOG) fed into a shallow classifier like an SVM. The pipeline was designed by humans; only the last stage was learned. Then a fast sequence of ImageNet (ILSVRC) results changed that:

The trend was obvious: deeper was better. VGG made this the explicit thesis, showing that stacking many small 3x3 filters beat using fewer large ones. So the natural question by 2015 was "is learning better networks as easy as stacking more layers?"

#Obstacle 1: vanishing and exploding gradients (mostly solved by 2015)

Gradients shrink or blow up multiplicatively as they backpropagate through many layers. By the time of this paper, this was largely handled by two things:

• Normalized initialization: Xavier/Glorot (2010) and He initialization (2015), the latter from the same author, tuned for ReLU networks.
• Batch Normalization (Ioffe and Szegedy, 2015), which normalizes layer inputs per mini-batch and keeps activations in a healthy range throughout training.

With these, very deep plain networks would at least converge. They just converged to something bad, which was the real puzzle.

#Obstacle 2: the degradation problem (the paper's actual target)

He et al. observed that once deep plain networks converged, accuracy saturated and then got worse with more layers. On CIFAR-10, a 56-layer plain CNN had higher error than a 20-layer one, and critically the training error was higher too. That rules out overfitting, which would show up as low training error and high test error. Something was wrong with optimization itself.

They pinned the paradox with a thought experiment. Take a trained shallow network and build a deeper one by appending extra layers that compute the identity function. This deeper network can, by construction, achieve exactly the shallow network's error. So a good solution always exists. If SGD cannot find a solution at least that good, the problem is that the solver struggles to learn identity mappings through a stack of nonlinear layers.

#The core idea

Rather than asking a stack of layers to directly fit a target function H(x), let it fit the residual F(x) = H(x) - x, and recover the target by adding the input back:

$$y=F(x,\{W_i\})+x$$

The addition is done element-wise via a shortcut (skip) connection that carries x forward unchanged. The hypothesis: if the optimal mapping is close to the identity, it is far easier for the solver to push the residual weights toward zero than to construct an identity mapping from scratch. Degenerating a block to a no-op becomes the default behavior instead of something that has to be learned.

Two properties made this immediately practical:

1. Identity shortcuts are parameter-free and add negligible compute, so a residual network has essentially the same cost as the plain network it is built from. Comparisons are apples-to-apples.
2. Nothing about training changes. Same SGD with backprop, no new losses, no auxiliary classifiers (which GoogLeNet needed), no gating.

#Handling dimension changes

Shortcuts only add cleanly when input and output shapes match. When a stage halves spatial resolution and doubles channels, the paper evaluates three options:

• (A) Identity shortcut with zero-padded extra channels. No parameters.
• (B) Projection shortcut, a 1x1 conv W_s x, used only where dimensions change.
• (C) Projection shortcut on every block.

B beat A slightly, C beat B marginally but added parameters and memory for little gain, so the released architectures use B. The takeaway is that the win comes from the shortcut topology, not from extra parameters.

#Block types

• Basic block (ResNet-18, ResNet-34): two 3x3 convs.
• Bottleneck block (ResNet-50, 101, 152): 1x1 conv to reduce channels, 3x3 conv, 1x1 conv to expand 4x. This keeps depth cheap and is why a 152-layer model is tractable.

Each conv is followed by BatchNorm, then ReLU, with the addition happening before the final ReLU of the block. Other inherited design rules: 3x3 convs, downsample by stride 2, double channel count when spatial size halves, and global average pooling instead of large fully-connected layers (borrowed from Network-in-Network and GoogLeNet).

Cost comparison worth remembering: ResNet-152 is about 11.3 GFLOPs with ~60M parameters, which is lower complexity than VGG-19 at ~19.6 GFLOPs, and VGG-16 alone carries 138M parameters. Deeper turned out to be cheaper.

#Results

• ILSVRC 2015 classification: 3.57% top-5 error with an ensemble, first place. A single ResNet-152 got about 4.49% top-5 on validation. Both are below the frequently cited ~5.1% human benchmark on this task.
• Clean ablation: plain-34 was worse than plain-18, reproducing degradation. ResNet-34 beat both, and beat plain-34 by roughly 3.5% top-1. Same architecture, shortcuts added, problem gone.
• Sweep of the same 2015 competition season: first place in all five tracks it entered, ImageNet classification, ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation. The detection wins came from dropping ResNet into Faster R-CNN as a backbone, which is what made the "general-purpose backbone" framing stick.
• CIFAR-10 depth stress test: a 110-layer ResNet reached 6.43% error. They also trained a 1202-layer network, which optimized without difficulty but overfit and landed at 7.93%, worse than the 110-layer model. The point was that the optimization barrier was removed, and what remained was ordinary overfitting on a small dataset.
• Response magnitude analysis: layer responses in residual networks were measurably smaller than in plain networks, which is the evidence they offer that blocks really are learning small perturbations near identity.

Training recipe, for reference: SGD with momentum 0.9, batch size 256, learning rate 0.1 divided by 10 on plateau, weight decay 1e-4, ~600k iterations, BatchNorm, no dropout, scale augmentation, 10-crop and fully convolutional multi-scale testing.

#Relation to prior work

Shortcut connections were not new. LSTM (1997) used a gated constant error carousel for the same reason, and Highway Networks (Srivastava, Greff, Schmidhuber, 2015) had already proposed shortcuts for very deep feedforward nets using learned gates:

$$y=H(x)\cdot T(x)+x\cdot (1-T(x))$$

The difference is that Highway gates are data-dependent and parametric, so they can close and cut off information, and in practice Highway Networks did not show accuracy gains at extreme depth. ResNet's shortcuts are ungated, parameter-free, and always open. All information is always preserved, and the residual branch can only add to it. Stripping out the gate is the whole trick.

#Why it still matters

• Direct successors: pre-activation ResNet ("Identity Mappings in Deep Residual Networks", He et al., ECCV 2016) reordered BN and ReLU to make the shortcut path a clean identity, which enabled a genuinely working 1001-layer CIFAR-10 model at 4.62% error. Then Wide ResNet (widen instead of deepen), ResNeXt (grouped convolutions), and DenseNet (concatenate instead of add).
• Residual connections became a default architectural primitive rather than a vision trick. The Transformer in "Attention Is All You Need" wraps every sublayer in a residual connection plus normalization, so every modern LLM depends on this idea.
• Later theory work explained why it works from other angles: residual nets behave like ensembles of many shallow paths (Veit et al., 2016), and skip connections visibly smooth the loss landscape (Li et al., 2018).
• ResNet-50 remains one of the most common backbones and benchmark baselines in the field, and the paper is among the most cited in modern science at well over 250k citations.

The one-line summary: the arc is AlexNet proving depth works, VGG pushing depth to ~19 layers, and ResNet removing depth as a constraint by changing what each block is asked to learn.