After Moore's Law, Performance Becomes a Software Problem

The End of Free Performance

For decades, software developers could expect new processors to make existing programs run faster without requiring major changes to the code. Improvements in transistor speed, processor architecture, clock frequency, and instruction-level parallelism provided a steady increase in computing performance. Between 1986 and about 2002, instruction-level parallelism together with faster transistors produced annual performance improvements of approximately 50%. That rate has since fallen sharply, with conventional processor performance improving by only a few percent per year in some measurements. ^[1]

This change has altered the relationship between hardware and software. Processor improvements cannot be treated as an automatic source of performance for existing applications anymore. New processors may provide more cores, wider vector units, larger memories, or specialized processing hardware, but software needs to use these resources effectively. This article discusses how the slowdown in processor scaling has changed the relationship between hardware and software, examining the role of software efficiency, parallelism, and hardware specialization in improving performance.

What Actually Changed?

Moore's Law and Dennard scaling are usually discussed together, but they describe different aspects of semiconductor development. Moore's Law describes the historical trend of increasing transistor density on integrated circuits. Dennard scaling describes how smaller transistors could operate at lower voltages and maintain approximately constant power density as transistor counts increased.

These trends worked together for many years. More transistors could be placed on a chip without creating a proportional increase in power density. Faster transistors also supported higher clock frequencies and greater single-thread performance.

Dennard scaling began to slow significantly around 2007 and was close to ending by about 2012.^{[1, 2]}  As voltage and power stopped scaling at the same rate as transistor density, increasing clock frequency also became difficult. Adding more transistors was still possible, but using all of them simultaneously at high frequency was constrained by power and thermal limits. Therefore, processor design changed direction. Instead of relying primarily on faster individual cores, manufacturers increased the number of cores and expanded other forms of parallel processing. Wider vector units allowed processors to operate on multiple data elements simultaneously. Larger and more complex memory systems were developed to keep processing units supplied with data.

Although this approach provides additional computing capacity, it also changes what software needs to do. A program that uses only one processor core cannot automatically benefit from a processor containing many cores. Similarly, a workload cannot take advantage of a vector unit or accelerator unless its computation can be expressed in a form that the hardware can execute efficiently.

Hardware Responds With Specialization

One response to the limitations of general-purpose processor scaling has been specialization. A general-purpose CPU is designed to execute a broad range of programs, which requires hardware mechanisms for handling different instruction patterns, branches, memory behavior, and operating conditions. A specialized accelerator can dedicate more of its resources to a narrower class of operations.

Modern accelerators can therefore use large numbers of parallel arithmetic units, specialized memory systems, lower-precision data types, and hardware designed around particular computational patterns. However, the obvious tradeoff is flexibility, as a specialized processor is useful when the workload matches the operations for which it was designed.

Google's first-generation Tensor Processing Unit (TPU) is an example of such specialization. It used a large matrix-multiplication unit and software-managed on-chip memory to accelerate neural-network inference. In its evaluation against contemporary CPUs and GPUs, an average speedup of about 15-30x and approximately 30-80x higher TOPS per watt was reported. ^[3] The broader analysis by Hennessy and Patterson cites a weighted comparison across six Google datacenter inference workloads in which the TPU achieved 29x the performance of a general-purpose CPU and more than 80x its energy efficiency. ^[1]

These numbers show what specialization can achieve when the hardware matches the workload. However, they also demonstrate its limitations. The TPU was effective because neural-network inference contained operations that could be mapped efficiently onto its architecture. Hardware specialization does not automatically improve every application. Software still determines whether a workload can take advantage of the available computational resources.

Desktop computer screen displaying program code. Source: AdobeStock.Software Efficiency Becomes a Performance Tool

In the earlier period of processor scaling, inefficient software could sometimes rely on faster hardware to reduce the practical impact of its inefficiencies. That did not make inefficient code equivalent to efficient code, but the processor performance could compensate for some of the difference from one generation to the next. The situation is different when hardware improvements provide smaller gains. Software efficiency then becomes one of the more direct ways to improve application performance.

In this regard, researchers from MIT illustrated the available optimization headroom with a 4096 × 4096 matrix-multiplication example. The researchers began with a Python implementation and progressively optimized it using a compiled language, parallel execution, memory-hierarchy optimization, vectorization, and Intel AVX instructions. The final implementation was reported to be 62,806 times faster than the original Python version, reducing execution time from roughly seven hours to 0.41 seconds. ^[2]

However, this example should not be interpreted as a typical improvement that developers should expect from rewriting Python code. The example is a deliberately optimized numerical workload and shows the amount of performance headroom that can exist in a poorly optimized implementation. It shows that the processor's theoretical capabilities and an application's actual performance can be very different.

Therefore, profiling is an important part of performance engineering. Developers can measure where execution time is spent, instead of optimizing code based on assumptions. The bottleneck might be an inefficient algorithm, excessive memory traffic, poor cache utilization, unnecessary computation, or insufficient parallelism. Once the bottleneck is known, optimization can focus on the part of the application that limits overall performance.

Parallelism Has to Be Designed In

Multicore processors and accelerators provide more computational resources, but an application cannot necessarily use all of them at once. Amdahl's Law describes this limitation by showing how the serial portion of a program restricts the maximum speedup available from parallel execution.

For instance, if a program has a portion that can only execute sequentially, adding more processor cores eventually provides diminishing returns. The parallel portion may become faster, but the serial portion is still a fixed part of the execution time. Hennessy and Patterson identify Amdahl's Law as one of the central challenges facing modern processor architecture.^[1] 

Parallelism is an architectural consideration for software, not simply an optimization added near the end of development. Data structures and algorithms may need to be designed so that independent operations can run simultaneously. Developers also need to manage synchronization, race conditions, communication between processing elements, and load imbalance. A workload divided across many cores can also lose some of its potential benefit if those cores spend too much time waiting for data or synchronizing with each other. Effective parallel software requires attention to both computation and communication.

Writing Software for Specialized Hardware

The growth of heterogeneous computing has created several programming approaches for using CPUs, GPUs, FPGAs, and other accelerators. For instance, CUDA provides a widely used programming model for NVIDIA GPUs, whereas OpenCL provides a more vendor-independent approach. OpenMP has also expanded from multicore CPU programming to accelerator offload. Similarly, SYCL provides another approach based on modern C++ and is designed to support heterogeneous processors through an open, cross-platform abstraction layer. 

SYCL shows an important distinction between source-code portability and performance portability. The same program can be written to target different hardware platforms, but identical source code does not guarantee identical performance. SYCL does not guarantee automatic or perfect performance portability because optimized kernels can differ between architectures. ^[4]

A 2023 study comparing CUDA and SYCL for a protein database search found broadly comparable performance on the tested NVIDIA GPUs. SYCL also provided portability across NVIDIA, AMD, and Intel GPUs. The same study showed that performance could still vary across architectures, and portable code may require architecture-specific tuning when performance is important. ^[5]

Therefore, a practical approach is to begin with optimized libraries and portable programming models where they provide sufficient performance. Hardware-specific kernels can then be introduced when profiling shows that a particular workload is limited by an operation that can benefit substantially from specialized treatment.

Performance Must Be Designed

The end of easy processor scaling has changed the way performance needs to be considered. Hardware can still provide substantial computing capacity through multicore architectures, vector processing, accelerators, and specialized processors. The difference is that software needs to expose workloads in a form that can use these resources effectively.

Performance is now determined by several connected factors. The algorithm determines how much work needs to be performed. Parallelism determines how much of that work can execute simultaneously. Memory behavior determines how efficiently data reaches the processing units. Hardware-aware software determines how effectively the available processor features are used.

Energy and computing cost are also part of this equation. A faster implementation that requires substantially more power may not provide the most useful result for a large-scale deployment. Domain-specific hardware can improve performance and energy efficiency when the workload matches the architecture, but software still determines how effectively that hardware is used. ^[2]

Hence, for engineers and software developers, the practical process starts with measurement. Profile a representative workload first. Identify where execution time, memory traffic, or synchronization overhead is spent. Then determine whether the bottleneck can be addressed through a better algorithm, parallel execution, optimized libraries, or specialized hardware. Now, performance improvements cannot simply be expected alone from the next processor generation, but they have to be designed into the software as well.

References

1. John L. Hennessy and David Patterson (2019) A New Golden Age for Computer Architecture. [Online] Communications of the ACM. Available at: https://cacm.acm.org/research/a-new-golden-age-for-computer-architecture/ (Accessed on September 13, 2026)

2. Leiserson, C. E., et al. (2020). There’s plenty of room at the Top: What will drive computer performance after Moore’s Law? https://www.science.org/doi/10.1126/science.aam9744 

3. Jouppi, N. P., et al. (2017, June). In-datacenter performance analysis of a tensor processing unit. In Proceedings of the 44th annual international symposium on computer architecture. https://arxiv.org/abs/1704.04760 

4. SYCL. [Online] Khronos. Available at:  https://www.khronos.org/sycl/ (Accessed on September 13, 2026)

5. Costanzo, M., et al. (2023). Comparing performance and portability between CUDA and SYCL for protein database search on NVIDIA, AMD, and Intel GPUs. In 2023 IEEE 35th International Symposium on Computer Architecture and High Performance Computing (SBAC-PAD). IEEE. https://arxiv.org/pdf/2309.09609