Sustainable data center cooling technologies

Sustainable data center cooling technologies and their optimized implementation for heat recovery and reuse

Feasibility study to analytically analyze use of nanofluids for cooling in co-location data center of 1.5 MW capacity and re-use the heat captured from data center for district heating purposes.

Data centers already account for 1.3 % of the world’s total electricity consumption in 2022 as per data gathered from IEA. This number is only expected to climb as the need for higher computational operations grows globally. Usually, a data center uses the most energy on its servers and cooling systems. Mostly, servers operate well below their maximum capacity making them energy inefficient. In current practices up to 40% of the power supplied is for cooling the IT equipment within the facility. Inefficient cooling methods lead to excessive energy consumption for cooling. Power usage effectiveness (PUE) is a helpful metric for determining data center energy efficiency. PUE is the ratio of power consumed by the entire facility to the power the IT equipment uses. A PUE of 1.0 indicates that all of the energy intake goes directly to the IT equipment. However, this is difficult to achieve for most data centers. Therefore, a PUE of 1.5 to 1.8 is considered to be effective for most facilities. That said, most data centers are likely operating at a PUE of 2.0 or above due to inefficient facility design, overcooling and poor management.

This feasibility study was carried out with an objective to investigate the improvements in efficiency that can be achieved with a 2 MW data center facility employing innovative cooling methods and reusing the captured energy from data centers for domestic and industrial heating purposes.

Synano performed this feasibility study for Deerns.

Deerns is an established engineering firm located in The Netherlands, that provides consultancy services for projects in multiple disciplines such as building systems and energy systems. Deerns has extensive experience in the design and development of data centers. Their data center group consists of dedicated and highly qualified specialists, with a combined data center engineering experience of over 500 man-years. Through this feasibility study, Deerns was looking to gain a competitive edge by providing their customers with cutting-edge consultancy to improve the energy efficiency of their data center facility. Further, the introduction of heat recovery and reuse can help Deerns become a player in the heat reuse market.

Project parameters

In order to determine the optimum cooling technology, a proposal for a pilot data center by Deerns was used with the original design being based on the liquid cooling technology. The research was thus based on modeling this data center with different liquid cooling technologies and then measuring the effect of nanofluids on each system, in terms of the increased heat transfer capability. Generally, the heat captured by cooling systems in data centers is discharged into the ambient using fans and chiller systems, making data centers less energy efficient. However, for this study we investigate liquid cooling technologies which also enables capturing and reusing the heat from the data center for domestic or industrial purposes. Thus, heat reuse is also considered for every alternative discussed.

A factor that becomes particularly important, especially for this case, is the Energy Reuse Factor (ERF). ERF is defined as the ratio of the energy captured and reused from the data center to the total energy consumption of the data center. The initial important parameters for the model are:

• Peak IT Load: 1650 kW
• Power density: 1.5 kW/m²
• Operating outside ambient temperatures: -13 to 31 °C
• Chip case temperature: 80-85 °C

Data centers typically operate at 25-40% of their design load. Given this limitation, the amount of reused heat is based on 25% of IT load to ensure heat is continuously delivered in typical load scenarios and accounts for a maximum of 412 kW assuming no heat losses and thermal resistances. However, this is not the case and the actual recovered heat will be lower than that.

Three modes of cooling were studied

1. CRAH/CRAC units

The heat is extracted from the data center by air recirculation using fans or air conditioning units (Computer Room Air Handling or Conditioning). The heat from the air is captured by water through an air-water heat exchanger and this heated water is pumped out of the IT environment where it can be reused.

2. Immersion cooling

With immersion cooling, the servers are completely submerged in a liquid and the liquid remains in a closed loop. Since the fluid is in direct contact with the electronics, it must be electrically non-conductive in nature. Usually, various forms of mineral oils or other hydrocarbon compounds are used. Here the non-conductive liquid rises in temperature as it takes the heat from the servers and releases this heat to water through a liquid-liquid heat exchanger. After cooling, the immersion fluid returns to the servers and heated water is pumped out for reuse.

3. Direct-to-chip cooling

It is a technology where the server heat is directly transferred to a coolant in a closed loop. In this technology, a metallic cold plate with coolant flowing inside is mounted on each server chip. Although the liquid is not in direct contact with the chip, most of the chip’s heat is transferred through conduction from the plate to the liquid. These types of cold plates can be designed to operate in either single or two phase. In single phase operation, the coolant remains in liquid phase and transfers heat from server to water through a liquid-liquid heat exchanger. The heated water can fulfil domestic or industrial energy needs. Two phase cooling devices such as thermosyphons, heat pipes, cold plates etc. are also used in such settings where the coolant absorbs heat from the chip to convert into vapor. This vapor is then condensed in a condenser unit transferring the heat to water.

For each case, a thermal resistance model was created. This model is analogous to an electrical circuit, where instead of a current, the heat flows through conduction or convection and all interfaces act as resistances. By decreasing the resistance, heat can be recovered at a higher temperature, which is valuable for heat reuse.

Results and conclusions

1. CRAH/CRAC units

In these setups, hot air outlet temperatures from IT equipment installed in data centers are 36°C, while cold inlet temperatures are kept at 24°C. The cooling water enters at 18 ° C and leaves after heating at 24°C. The resistance to heat transfer in this scenario is heavily dominated by the small convective heat transfer coefficient of air. This is due to the low specific heat and thermal conductivity of air compared to water. This makes air the real limiting factor to any increase in heat transfer. Therefore, changing the coolant will not have any impact on the overall heat transfer. Thus, nanofluids are not a feasible solution for implementation in this scenario. The ‘hot’ water extracted remains at 24° C. This is low grade heat and is unsuitable for applications such as desalination, absorption refrigeration, space and water heating etc.

2. Immersion cooling

A single immersion cooling module is capable of cooling up to 10 kW. The oil used for immersion is usually a standard mineral oil. In the setup, the immersion oil takes the heat from the servers to reach a temperature of 63°C. This heat is transferred to the coolant supplied at 40° C through a liquid-liquid heat exchanger. After the heat is transferred, the coolant comes out at 50° C while the oil returns to the setup at a temperature of 45° C. In such a setup, the nanofluids replace the coolant in the secondary loop.

3. Direct-to-chip cooling

Each server consists of 200 W per chip with maximum chip junction temperature to be maintained at 80 degC. The resistance from the chip case to the coolant was lower, with the coolant in this loop attaining a 53° C temperature. Finally, the heat from coolant is transferred to the water in the secondary loop (for heat recovery) with hot water at 50°C available for reuse.

By replacing the coolant in the loop with nanofluids, a reduction in thermal resistance of 20% was calculated. This increased the coolant output temperatures by 2 degC and resulted in 9% pump power reduction. Another method for using nanoparticles in this system was to make nano-modifications on the surface of the cold plate by using nanoparticle coatings. This would result in about 50-60% reduction in thermal resistance leading to a 5 degC increase in coolant output temperature and reduction of about 25% pump power in primary loop. This temperature rise could be valuable for any heat reuse applications and hence proved technical feasibility of employing nanoparticles for improving cooling performance in data centers.

For both cases-2 and 3, two phase variants of the system are also possible in which nanofluids and nano surface modifications would give a considerable improvement. However, the scope of this study was limited to the use of single-phase liquid cooling. The PUE and ERF values were calculated for all three cases as shown in table below:

Since improvements with nanofluids are not significant in case-1 and case-2, the improvements that can be provided for case-3 are shown below:

With this study, it becomes clear that nanofluids and nanoparticle surface modifications are feasible solutions to improve heat transfer and hence PUE of data centers when employed in Direct-to-chip cooling method.