Luigi Brochard was a Distinguished Engineer at IBM, then at Lenovo Data Center Group where he designed energy aware software systems. He is now a consultant on energy efficiency.Vinod Kamath worked with IBMx86 as a thermal engineer and is now a Senior Technical Staff Member with the Lenovo Data Center Group, where he is responsible for the water-cooled server design.Julita Corbalán is Associate Professor at the UPB, associate researcher at BSC-CNS and technical leader for the EAR Lenovo-BSC project.Scott Holland is a thermal engineer with Lenovo Data Center Group and is responsible for providing direct liquid-cooling solutions. His most recent project was the Lenovo ThinkSystem SD650.Walter Mittelbach has been working on adsorption chillers at Fraunhofer ISE since 1995. He also founded two companies: Fahrenheit GmbH (2002) and Sorption Technologies GmbH (2018).Michael Ott is a senior researcher in the HPC division at Leibniz Supercomputing Centre, where he leads the research activities on energy efficiency.

• Introduction ixChapter 1. Systems in Data Centers 11.1. Servers 11.2. Storage arrays 31.3. Data center networking 41.4. Components 51.4.1. Central processing unit 51.4.2. Graphics processing unit 71.4.3. Volatile memory 81.4.4. Non-volatile memory 101.4.5. Non-volatile storage 101.4.6. Spinning disks and tape storage 131.4.7. Motherboard 151.4.8. PCIe I/O cards 161.4.9. Power supplies 171.4.10. Fans 18Chapter 2. Cooling Servers 192.1. Evolution of cooling for mainframe, midrange and distributed computers from the 1960s to 1990s 192.2. Emergence of cooling for scale out computers from 1990s to 2010s 202.3. Chassis and rack cooling methods 232.4. Metrics considered for cooling 272.4.1. Efficiency 272.4.2. Reliability cost 282.4.3. Thermal performance 292.5. Material used for cooling 312.6. System layout and cooling air flow optimization 32Chapter 3. Cooling the Data Center 373.1. System cooling technologies used 373.2. Air-cooled data center 383.2.1. Conventional air-cooled data center 383.3. ASHRAE data center cooling standards 403.3.1. Operation and temperature classes 403.3.2. Liquid cooling classes 413.3.3. Server and rack power trend 423.4. Liquid-cooled racks 433.5. Liquid-cooled servers 463.5.1. Water heat capacity 463.5.2. Thermal conduction module 473.5.3. Full node heat removal with cold plates 483.5.4. Modular heat removal with cold plates 503.5.5. Immersion cooling 503.5.6. Recent DWC servers 513.6. Free cooling 543.7. Waste heat reuse 543.7.1. Reusing heat as heat 543.7.2. Transforming heat with adsorption chillers 55Chapter 4. Power Consumption of Servers and Workloads 654.1. Trends in power consumption for processors 654.1.1. Moore’s and Dennard’s laws 684.1.2. Floating point instructions on Xeon processors 724.1.3. CPU frequency of instructions on Intel Xeon processors 734.2. Trends in power consumption for GPUs 744.2.1. Moore’s and Dennard’s laws 774.3. ACPI states 784.4. The power equation 83Chapter 5. Power and Performance of Workloads 875.1. Power and performance of workloads 875.1.1. SKU power and performance variations 875.1.2. System parameters 895.1.3. Workloads used 925.1.4. CPU-bound and memory-bound workloads 925.1.5. DC node power versus components power 935.2. Power, thermal and performance on air-cooled servers with Intel Xeon 945.2.1. Frequency, power and performance of simple SIMD instructions 955.2.2. Power, thermal and performance behavior of HPL 985.2.3. Power, thermal and performance behavior of STREAM 1035.2.4. Power, thermal and performance behavior of real workloads 1075.2.5. Power, thermal and frequency differences between CPUs 1155.3. Power, thermal and performance on water-cooled servers with Intel Xeon 1245.3.1. Impact on CPU temperature 1245.3.2. Impact on voltage and frequency 1255.3.3. Impact on power consumption and performance 1275.4. Conclusions on the impact of cooling on power and performance 131Chapter 6. Monitoring and Controlling Power and Performance of Servers and Data Centers 1336.1. Monitoring power and performance of servers 1336.1.1. Sensors and APIs for power and thermal monitoring on servers 1346.1.2. Monitoring performance on servers 1406.2. Modeling power and performance of servers 1426.2.1. Cycle-accurate performance models 1426.2.2. Descriptive models 1426.2.3. Predictive models 1446.3. Software to optimize power and energy of servers 1496.3.1. LoadLeveler job scheduler with energy aware feature 1506.3.2. Energy Aware Runtime (EAR) 1516.3.3. Other run time systems to manage power 1536.4. Monitoring, controlling and optimizing the data center 1546.4.1. Monitoring the data center 1546.4.2. Integration of the data center infrastructure with the IT devices 156Chapter 7. PUE, ERE and TCO of Various Cooling Solutions 1597.1. Power usage effectiveness, energy reuse effectiveness and total cost of ownership 1597.1.1. Power usage effectiveness and energy reuse effectiveness 1597.1.2. PUE and free cooling 1627.1.3. ERE and waste heat reuse 1637.2. Examples of data centers PUE and EREs 1647.2.1. NREL Research Support Facility, CO 1647.2.2. Leibnitz Supercomputing data center in Germany 1667.3. Impact of cooling on TCO with no waste heat reuse 1737.3.1. Impact of electricity price on TCO 1777.3.2. Impact of node power on TCO 1787.3.3. Impact of free cooling on TCO 1817.4. Emerging technologies and their impact on TCO 1837.4.1. Waste heat reuse 1847.4.2. Renewable electricity generation 1897.4.3. Storing excess energy for later reuse 1927.4.4. Toward a net-zero energy data center 193Conclusion 195References 199Index 209