In AI systems training, a report has it that over a seven-year period algorithmic efficiency has increased three times faster than Moore’s Law.

If you are or have been a programmer then you know that there are several different ways to turn a functional spec into a computer program. And they can differ a lot in terms of computing resources usage.

Imagine a simple sort. If you code the program naïvely, as the number of elements to be sorted grows the amount of CPU and memory employed will increase like the square of that number. Using Tony Hoare’s Quicksort instead, the cost increase will only be just a little bigger than the number of elements themselves -definitely not its square.

An ill-advised executive would support the dumber programmer by pouring in more hardware as the sizes of sorting lists increase. But it may be cheaper to hire the smarter programmer while keeping the same hardware.

The new finding

Assessing computing performances is a difficult art. And separating algorithmic performances from the overall picture is even more challenging.

However, researchers at OpenAI have issued the preliminary results of an analysis showing that in 2012-2018 the amount of computing needed to train a neural network to the same performance at a specific task has decreased 44 times. By contrast, Moore’s Law would yield an 11x cost improvement over the same period.

The results suggest that “for AI tasks with high levels of recent investment, algorithmic progress has yielded more gains than classical hardware efficiency”.

The authors generalized their conceptual experiment and concluded that, conservatively, one can think that algorithmic improvement has contributed >3 times than Moore’s Law over the period of time examined and for the largest AI training tasks.

Make no mistake: putting in more hardware (yellow area in the Figure) has by far been the biggest factor in improving AI systems in the past decade. But the influence of algorithmic efficiency has been growing faster than that of hardware.

Given a fixed performance goal, the payback of a better algorithm is -on average- about 3 times bigger than that of a more powerful computer.

So while hardware is going to keep playing the largest part in the development of ever-more-powerful AI systems, it is time to focus on programming skills as well.

So What?

Since hardware costs have been declining exponentially for half a century, the issue of programming skills typically only comes up in extreme program-complexity situations.

Training of deep-learning systems is quickly becoming one of these. (See for example here if you want to know of the challenges in deciding which algorithm to pick for an optimization problem typically arising in machine learning).

And although selecting better people is more difficult than purchasing more-powerful machines, for both AI vendors and large users now is the time to start learning the game.

It is convenient to pick programmers, data analysts, and machine-learning engineers based on their mathematical and programming optimization skills. Most of the time, it will save you more money than selecting the best hardware.

Engineering tools are growing and maturing, including aids to evaluate the complexity of a machine-learning algorithm. But there still is no substitute for human decisions at that stage of design.


I am the author of this article and it expresses my own opinions. I have no vested interest in any of the products, firms, or institutions mentioned in this post.