Cooling Technologies

Cooling is now the binding constraint on AI compute density. As GPU power consumption has increased from 300W (A100) to 700W (H100) to 1000W+ per chip (B200), the ability to remove heat has become the limiting factor in how many GPUs you can pack into a given space. The cooling technology frontier is moving faster than it has in decades.

Air Cooling - Still Common, Running Out of Headroom

Traditional data centers use air cooling: cold air flows from raised floor tiles through server racks, picks up heat, and returns as hot air to be chilled by CRAC (Computer Room Air Conditioning) units:

Cold aisle / Hot aisle containment:
  ┌──────────────────────────────────────┐
  │ HOT AISLE (hot exhaust from servers) │
  ├──────────────────────────────────────┤
  │ SERVER RACK  (front→back airflow)    │
  ├──────────────────────────────────────┤
  │ COLD AISLE (cold supply air)         │
  └──────────────────────────────────────┘

Limitations:
  Air has very low thermal capacity (heat absorption)
  Effective for: up to ~25 kW/rack
  Usable for AI: small inference servers, CPU-based ML
  Not viable for: modern H100/B200 GPU racks

Air cooling is cheap and proven, but maxes out around 25 kW/rack. An 8× H100 DGX server needs 10 kW and already pushes air cooling near its limits. An NVIDIA GB200 NVL72 rack at 120+ kW is impossible to air-cool.

Direct Liquid Cooling (DLC)

DLC routes chilled water directly to cold plates mounted on GPUs and CPUs. The liquid absorbs heat far more efficiently than air (water has ~3,500× the heat capacity of air by volume):

• Cold plates mounted directly on GPU die or heat spreader
• Chilled water (typically 25–45°C supply) circulates through the plates
• Rear-door heat exchangers or in-row cooling units reject heat from the water loop to the building chiller
• Handles 40–100 kW/rack reliably
• NVIDIA DGX H100 and H200 systems ship DLC-ready
• Does not eliminate air cooling entirely - some components (memory, power supplies, motherboard) still need air

Immersion Cooling

The most capable cooling technology: submerge entire servers in dielectric fluid (a non-conductive liquid that doesn't damage electronics). The fluid absorbs heat and carries it to a heat exchanger:

Immersion cooling can handle 250+ kW/tank and requires no air cooling at all. Advantages: complete silence (no fans), higher component reliability (no vibration, no humidity variation), smaller facility footprint. The challenge: servicing servers requires removing them from the fluid tank, which is messy and slower than air-cooled rack service.

Rear-Door Heat Exchangers

A compromise between air and full liquid cooling: a door-sized heat exchanger mounts on the back of a standard server rack. Chilled water runs through the door; air from the servers passes through it and is cooled before re-entering the room. Handles 30–50 kW/rack with no rack modification. Used as an upgrade path for existing air-cooled data centers.

NVIDIA's GB200 Cooling Requirements

The NVIDIA GB200 NVL72 "rack" (72 GPUs + 36 Grace CPUs in a single rack system) illustrates where the industry is heading:

• 120–140 kW total rack power
• Requires liquid cooling - NVIDIA ships the system with integrated liquid cooling; no air-cooled variant
• Copper manifold runs to each GPU; factory-integrated coolant distribution unit (CDU)
• Data centers must have liquid cooling infrastructure in place before GB200 clusters can be deployed
• This requirement has delayed many GB200 deployments as operators retrofit facilities

Industry Trends

• All major AI hardware vendors (NVIDIA, AMD, Intel) are designing future chips that assume liquid cooling - not just support it optionally
• New AI data centers being built in 2025–2027 are designed from the ground up for liquid cooling infrastructure
• Existing colocation facilities are rapidly losing relevance for AI training workloads - they cannot support the density
• Cooling infrastructure lead time (12–18 months for a new liquid-cooled facility) is now a competitive advantage for cloud providers who built early