The hardware that made this possible was designed for drawing pictures

A different shape of processor, built for a different job

A conventional processor is optimised to run one sequence of instructions as fast as possible, with elaborate machinery for predicting branches and managing dependencies. It is designed for work where each step may depend on the one before, which describes most software ever written.

Rendering three-dimensional graphics is not that kind of work. Each pixel and each vertex can be computed largely independently of the others, using the same arithmetic applied to different values. So graphics hardware evolved in the opposite direction: many simple arithmetic units running in parallel, fed by very wide memory access.

The arithmetic itself is nothing exotic. Transforming geometry and shading surfaces amounts to multiplying vectors by matrices, in enormous quantity, at moderate precision. Individual accuracy hardly matters when the result is a coloured pixel that will be looked at for a sixtieth of a second, so the hardware was built to do a great deal of approximate arithmetic rather than a little of the exact kind.

Neural networks want the same operation

The core computation in a neural network is also a matrix multiplication, repeated layer after layer. Every unit in a layer computes a weighted sum of the previous layer’s outputs, and all those sums are independent of one another, so they can be computed simultaneously.

That is precisely the workload graphics hardware was built for. Moderate precision is acceptable in both cases, the parallelism is abundant in both cases, and the memory access patterns are regular in both cases. The match is close enough that it looks designed, and it was not.

The consequence was a step change in what was affordable. Training runs that would have taken an impractical amount of time on conventional processors became feasible, which meant ideas could be tested rather than merely argued about.

The bridge was a programming interface, not a new chip

Early attempts to use graphics hardware for general computation involved disguising the calculation as a rendering task, which was ingenious and painful. What changed the situation was the arrival, in the second half of the 2000s, of interfaces that let developers write ordinary programs for these processors without pretending to draw anything.

That is a software development rather than a hardware one, and it deserves more credit than it gets in the usual telling. The capability had existed for years; what was missing was a route to it that researchers rather than graphics specialists could take.

Once the route existed, groups working on neural networks adopted it quickly, because the fit was so obvious once anyone tried it. Within a few years, results that shifted the field’s direction were being produced on hardware sold for video games.

A demand-side accident as much as a supply-side one

The economics matter here. Graphics processors were developed and refined because a very large consumer market wanted better games, and that market funded successive generations of increasingly capable parallel hardware over decades.

No research programme in machine learning could have paid for that development. The field inherited an enormously expensive piece of industrial capability that had been built for entirely unrelated reasons, and it inherited it at consumer prices. A laboratory could buy a handful of cards, assemble them into something modest, and run experiments that would previously have required time on a shared national facility.

This is worth pausing on, because narratives of technological progress usually run from insight to application. Here a substantial part of the causation ran the other way: the hardware existed, and its existence made a set of previously impractical ideas worth revisiting.

What the episode suggests about how progress arrives

The underlying algorithms had been published long before. What changed was that the arithmetic became cheap enough to run them at a scale where they worked, and the field’s subsequent trajectory follows the cost of computation more closely than it follows the sequence of ideas.

That reading can be pushed too far. Plenty of genuine algorithmic innovation was required to use the hardware effectively, and treating the whole story as a hardware accident does a disservice to work that was neither obvious nor easy.

The moderate conclusion is that ideas and capability arrive on separate schedules, and the field advances when they happen to meet. Which implies something uncomfortable about the ideas currently sitting in old papers, waiting for the machine that makes them affordable.