The Intel Haswell-EP processor generation introduces several major advancements of power control and energy-efficiency features. For computationally intense applications using advanced vector extension instructions, the processor cannot continuously operate at full speed but instead reduces its frequency below the nominal frequency to maintain operations within thermal design power (TDP) limitations. Moreover, the running average power limitation (RAPL) mechanism to enforce the TDP limitation has changed from a modeling to a measurement approach. The combination of these two novelties have significant implications. Through measurements on an Intel Sandy Bridge-EP cluster, we show that previous generations have sustained homogeneous performance across multiple CPUs and compensated for hardware manufacturing variability through varying power consumption. In contrast, our measurements on a Petaflop Haswell system show that this generation exhibits rather homogeneous power consumption limited by the TDP and capped by the improved RAPL while providing inhomogeneous performance under full load. Since all of these controls are transparent to the user, this behavior is likely to complicate performance analysis tasks and impact tightly coupled parallel applications.
@cite_0: Saving energy and, therefore, reducing the Total Cost of Ownership (TCO) for High Performance Computing (HPC) data centers has increasingly generated attention in light of rising energy costs and the technical hurdles imposed when powering multi-MW data centers. The broadest impact on data center energy efficiency can be achieved by techniques that do not require application specific tuning. Improving the Power Usage Effectiveness (PUE), for example, benefits everything that happens in a data center. Less broad but still better than individual application tuning would be to improve the energy efficiency of the HPC system itself. One property of homogeneous HPC systems that hasn’t been considered so far is the existence of node power variation.
@cite_13: 
@cite_12: The last decade has seen power consumption move from an afterthought to the foremost design constraint of new supercomputers. Measuring the power of a supercomputer can be a daunting proposition, and as a result, many published measurements are extrapolated. This paper explores the validity of these extrapolations in the context of inter-node power variability and power variations over time within a run. We characterize power variability across nodes in systems at eight supercomputer centers across the globe. This characterization shows that the current requirement for measurements submitted to the Green500 and others is insufficient, allowing variations of up to 20 due to measurement timing and a further 10--15 due to insufficient sample sizes. This paper proposes new power and energy measurement requirements for supercomputers, some of which have been accepted for use by the Green500 and Top500, to ensure consistent accuracy.
@cite_8: Processor stress test utilities are important tools for a number of different use cases. In particular, cooling systems need to be tested at maximum load in order to ensure that they fulfill their specifications. Additionally, a test system characterization in terms of idle and maximum power consumption is often a prerequisite for energy efficiency research. This creates the need for a simple yet versatile tool that generates near-peak power consumption of compute nodes. While in different research areas tools such as LINPACK and Prime95 are commonly used, these tools are just highly optimized and compute intense routines that solve specific computational problems. As stress test utilities they are unnecessarily hard to use and in many cases unreliable in terms of power consumption maximization. We propose FIRESTARTER, an Open Source tool that is specifically designed to create near-peak power consumption. Our experiments show that this task cannot be achieved with generic high-level language code. We therefore use highly optimized assembly routines that take the specific properties of a given processor microarchitecture into account. A study on four compute nodes with current or last generation x86_64 processors shows that we reliably exceed the power consumption of other stress tests and create very steady power consumption patterns.
We discuss power and performance variations of two generations of Intel processors in this paper. describe node-level power variations for more than 260 nodes equipped with AMD Magny Cours executing mprime @cite_13 and Intel Sandy Bridge processors executing FIRESTARTER @cite_8 in @cite_0 . They propose to change the batch systems partition to increase the utilization of less power consuming nodes and decrease the utilization of inefficient ones. describe power consumption variations over time and between nodes for several systems based on different processors (Intel Nehalem-EP, Nehalem-EX, Sandy Bridge-EP, and AMD FirePro GPUs) with a focus on improving large-scale power measurement methodologies @cite_12 .