Part 5 · 2026

What it costs to train ResNet today

A result that took a research team weeks of cluster time in 2015 is now a same-day errand for about twenty dollars. Plus: how slow is a Mac, really?

Source: notes/05-resnet-training-cost-today.md · 2,187 words · 2026

An estimate of the compute needed to reproduce ResNet-50 on ImageNet in 2026, what that costs to rent, and how long it would take on Apple Silicon. Background in ResNet.

Headline numbers, for the impatient:

• The full job is about 1.4 exa multiply-adds, which is 2.8 ExaFLOPs if you count each multiply-add as two floating point operations. Both conventions are in common use and they differ by exactly 2x, which is worth pinning down before comparing anyone's numbers to anyone else's.
• Renting a cloud GPU costs roughly $20 and takes 8 to 13 hours on a single A100 or H100.
• On a Mac M4 it is feasible but slow: roughly 6 to 8 days on an M4 Max, 12 to 16 days on an M4 Pro, and 3 to 4 weeks on a base M4.
• You almost certainly should not do this on a Mac, and the last section covers what to run locally instead.

#Defining the job

The standard reproduction target is ResNet-50 on ImageNet-1k to about 76% top-1 accuracy:

A note on units, since this trips up every comparison. The ResNet paper says ResNet-50 is "3.8 billion FLOPs" and means multiply-adds; most later work says 4.1 GFLOPs and also means multiply-adds, the difference being whether the downsample convolutions and classifier are counted. Neither means true floating point operations, which are 2x higher because a multiply-add is two of them. This note quotes multiply-adds and gives the doubled figure alongside.

For reference, ResNet-152 is about 11.3 billion multiply-adds forward, so the same recipe on the deepest variant is roughly 2.8x this.

#Cloud GPUs: about $20 and part of a day

Using NVIDIA's own published mixed-precision throughput for ResNet-50 v1.5, plus independent measurements from QMUL:

The interesting thing about that last column is that it barely moves. Scaling out buys wall-clock time, not savings, because you are purchasing a fixed quantity of arithmetic either way. Parallel efficiency is good enough on this workload (about 6.7x to 7.7x on 8 GPUs) that the cost stays flat. Rental rates vary by a factor of two or three between hyperscalers and spot or community providers, so the honest range is $10 to $50.

For an energy anchor: an MLPerf Training v4.0 submission on 8x H100 SXM hit the accuracy target in 13 minutes 21 seconds, drawing 6.89 kW, for a total of 1.53 kWh. That is about 20 cents of electricity.

#Mac M4: grounding the estimate in measured data

Peak FLOPS is the wrong place to start for Apple Silicon, because PyTorch's MPS backend achieves a low fraction of it. Better to anchor on real measurements. Daniel Bourke's Apple Silicon benchmark suite trained ResNet-50 on Food101 (75,750 images at 224x224, fp32, PyTorch 2.1) and published raw per-epoch times, which convert directly to throughput:

Two things to note about this data. First, it is fp32, because mixed precision was not usable on MPS at the time of measurement; bf16 on MPS has since improved, which should be worth something like 1.3x. Second, the per-epoch times are nearly flat across batch sizes 32, 64, and 128 on the M3 Max, which suggests the pipeline was at least partly bound by JPEG decode rather than by the GPU. Apple Silicon guidance typically recommends num_workers=0, which makes this worse.

#Sanity-checking the measurements against peak

Before extrapolating, it is worth confirming these numbers are physically sensible. At 12.3 billion multiply-adds per training image, or 24.6 GFLOPs:

The Apple chips cluster tightly at 23% to 28%, and the V100 is not far off at 20%, which is reassuring: the measurements are consistent with each other and with what a convolutional training workload typically extracts from a GPU. The TITAN RTX outlier at 41% most likely reflects a better-fed data pipeline on that machine. Since Apple Silicon lands in a narrow band, extrapolating by compute ratio at a fixed 25% utilization is defensible.

#Scaling to M4

The M4 family's peak FP32 figures:

Holding utilization at 25% of peak gives the fp32 figure; multiplying by 1.3 for bf16 autocast gives the fast end of each range.

Per epoch, that is roughly 1.6 to 2 hours on an M4 Max, 3 to 4 hours on an M4 Pro, and 6 to 8 hours on a base M4. The bf16 multiplier is the softest assumption in the table; if MPS does not deliver it for your model, use the slow end.

#Practical caveats that make it worse

• Thermal throttling. These are multi-day sustained loads. A MacBook Pro will not hold peak clocks for six days; a Mac Studio or Mac Mini with better cooling does much better. Add 20% to 40% to laptop estimates.
• The machine is unusable meanwhile. Unified memory means the training job is competing with your OS and apps for both memory and bandwidth.
• Memory limits batch size. ResNet-50 at batch 256 in fp32 wants roughly 20 to 25 GB of activations, about 12 to 15 GB with mixed precision. A 16 GB base M4 is limited to something like batch 64 to 96, which hurts both throughput and the accuracy of the standard recipe, since the published learning rate schedule assumes batch 256.
• Data pipeline. Sustaining 200+ images/sec of JPEG decode plus augmentation needs real CPU work alongside the GPU job.
• Reliability over days. A week-long run needs checkpointing and a machine that does not sleep, update, or get closed.

#Energy, which is the one place Apple wins

An M4 Max under sustained GPU load draws roughly 50 to 90 W at the wall. Seven days at 70 W is about 12 kWh, or a dollar or two of electricity. A single A100 finishing in 13 hours at 400 W for the GPU alone is around 5 kWh, and more once you count the host. So total energy is within a small factor, and the Mac is not obviously worse on a per-job basis. It just spreads comparable energy over 10 to 15 times the wall-clock time.

#Historical comparison

The original 2015 runs used 8 GPUs of the Tesla K40 era, roughly 4 to 5 TFLOPS FP32 each with no tensor cores, for something like two to three weeks per deep model. ResNet-152 under the paper's schedule of 600k iterations at batch 256 works out to about 5.2 exa multiply-adds, or 10.4 ExaFLOPs, roughly 3.7x the ResNet-50 job costed above.

So an M4 Max is in the same league as the entire 8-GPU server that produced the original paper, and a single rented H100 does the ResNet-50 job in under eight hours for the price of lunch. A result that took a Microsoft Research team weeks of cluster time is now a same-day errand.

#What you should actually run locally

Reproducing ImageNet on a Mac is a bad use of a week. These are all genuinely practical:

The CIFAR estimates carry a large error bar in the optimistic direction. 32x32 batches leave most of the GPU idle, so the utilization assumption above does not hold; expect to land toward the slow end.

The CIFAR-10 option is the one I would pick to actually learn something from this paper, because it reproduces the interesting experiment rather than the headline one. The degradation result described in The challenge, where a 56-layer plain network has higher training error than a 20-layer one, is a CIFAR-10 experiment. You can train plain-20, plain-56, ResNet-20, and ResNet-56 in an afternoon on a Mac and watch the residual connections fix a failure you produced yourself. That is a far better return on a day of compute than a 76.1% top-1 number you could have downloaded.

If you do want the full ImageNet run, rent one A100 for a day. It costs less than the electricity plus the week of degraded laptop performance.

#If you do run it on the Mac anyway

• Use bf16 autocast, which current MPS supports, rather than fp32.
• Consider MLX instead of PyTorch MPS. It is designed around unified memory and often does better on Apple Silicon, though the vision training ecosystem around it is thinner.
• Use a modern fast recipe rather than the 2015 one. FFCV-style pipelines with progressive resizing, starting at 160px and ending near 192px, reach comparable accuracy in far fewer epoch-equivalents, which can cut a multi-day estimate by half or more.
• Pre-resize the dataset to shorter-side 256 and store it as raw or a packed format, so JPEG decode stops being the bottleneck.
• Checkpoint every epoch, and run it on a desktop Mac, not a laptop.

#Sources and assumptions

• Compute model: ResNet-50 at 4.1 billion multiply-adds forward, backward at 2x forward. This is the standard convention and is accurate to maybe 10%. True FLOPs are 2x these figures.
• NVIDIA DeepLearningExamples ResNet-50 v1.5 published throughput: 2,470 img/s on 1x A100 with AMP, quoted at about 14 hours for 90 epochs, which matches the arithmetic here.
• QMUL HPC measurements: 2,451 img/s on A100 PCIe, 3,147 on H100 PCIe, 4,380 on H100 SXM.
• Apple Silicon throughput derived from mrdbourke/mac-ml-speed-test raw per-epoch CSVs for ResNet-50 on Food101 at 224x224.
• M4 peak FLOPS from published Apple silicon specifications.
• Cloud pricing assumed at $1.30/hr per A100 and $2.50/hr per H100, which is mid-market as of 2026. Spot and community providers run meaningfully cheaper.

The M4 numbers are extrapolations from M3 measurements at a fixed 25% utilization, not direct measurements, so treat them as accurate to within roughly a factor of 1.5.