Sunday, April 9, 2023

The Great CPU Stagnation

For at least five decades, Moore's law consistently delivered increasing numbers of transistors. Equally significant, Dennard scaling led to each transistor using less energy, enabling higher clock frequencies. This was great, as higher clock frequencies enhanced existing software performance automatically, without necessitating any code rewrite. However, around 2005, Dennard scaling began to falter, and clock frequencies have largely plateaued since then.

Despite this, Moore's law continued to advance, with the additional available transistors being channeled into creating more cores per chip. The following graph displays the number of cores for the largest available x86 CPU at the time:
Notice the logarithmic scale: this represents the exponential trend we had become accustomed to, with core counts doubling roughly every three years. Regrettably, when considering cost per core, this impressive trend appears to have stalled, ushering in an era of CPU stagnation.

To demonstrate this stagnation, I gathered data from on AMD's Epyc single-socket CPU lineup, introduced in 2017 and now in its fourth generation (Naples, Rome, Milan, Genoa):

Model Gen Launch Cores GHz IPC Price
7351P Naples 06/2017 16 2.4 1.00 $750
7401P Naples 06/2017 24 2.0 1.00 $1,075
7551P Naples 06/2017 32 2.0 1.00 $2,100
7302P Rome 08/2019 16 3.0 1.15 $825
7402P Rome 08/2019 24 2.8 1.15 $1,250
7502P Rome 08/2019 32 2.5 1.15 $2,300
7702P Rome 08/2019 64 2.0 1.15 $4,425
7313P Milan 03/2021 16 3.0 1.37 $913
7443P Milan 03/2021 24 2.9 1.37 $1,337
7543P Milan 03/2021 32 2.8 1.37 $2,730
7713P Milan 03/2021 64 2.0 1.37 $5,010
9354P Genoa 11/2022 32 3.3 1.57 $2,730
9454P Genoa 11/2022 48 2.8 1.57 $4,598
9554P Genoa 11/2022 64 3.1 1.57 $7,104
9654P Genoa 11/2022 96 2.4 1.57 $10,625

Over these past six years, AMD has emerged as the x86 performance per dollar leader. Examining these numbers should provide insight into the state of server CPUs. Let's first observe CPU cores per dollar:

This deviates significantly from the expected exponential improvement graphs. In fact, CPU cores are becoming slightly more expensive over time! Admittedly, newer cores outperform their predecessors. When accounting for both clock frequency and higher IPC, we obtain the following image:

This isn't much better. The performance improvement over a 6-year period is underwhelming when normalized for cost. Similar results can also be observed for Intel CPUs in EC2.

Lastly, let's examine transistor counts, only taking into account the logic transistors. Despite improved production nodes from 14nm (Naples) over 7nm (Rome/Milan) to 5nm (Genoa), cost-adjusted figures reveal stagnation:

In conclusion, the results are disheartening. Rapid and exponential improvements in CPU speed seem to be relics of the past. We now find ourselves in a markedly different landscape compared to the historical norm in computing. The implications could be far-reaching. For example, most software is extremely inefficient when compared to what hardware can theoretically achieve, and maybe this needs to change. Furthermore, historically specialized chips enjoyed only limited success due to the rapid advancement of commodity CPUs. Perhaps, custom chips will have a much bigger role in the future.

P.S. Due to popular demand, here's how the last graph looks like after adjusting for inflation: