TCS and FWS: data center cooling loop architecture and scope splits

Why the loops are separated

The loops are separated because their requirements are incompatible.

The server loop is small, closed and tight. Its fluid passes through the narrow channels of the cold plates, so the cleanliness requirement is strict and the permitted pressure low. The facility loop is large, it reaches outdoors, and its conditions vary with ambient temperature. Its water quality depends on local conditions and on how heat is rejected.

If the loops were combined, facility loop pressure, contamination and temperature variation would pass straight into the servers. A CDU is therefore first of all a separator, and only secondly a heat transfer device. Its operation is covered in more detail in the article on CDUs in the data center.

How the loops differ

TCS / technology cooling system: serves IT cooling components. Its fluid, materials, pressure and cleanliness follow OEM requirements.

FWS / facility water system: connects CDUs to heat rejection or recovery. Design conditions depend on the plant.

Pressure: neither loop is automatically specified below the other. A heat-exchanger leak flows from higher to lower pressure.

Scope split: agree connections, power, controls, flushing, filling, tests, acceptance and warranty requirements.

Temperature classes decide the economics

ASHRAE’s TC9.9 committee defines W classes W17, W27, W32, W40, W45 and W+ for the facility loop, where the number is the upper limit of supply water temperature in degrees Celsius. Corresponding S classes apply to the equipment-side server loop.

Choosing the class is the single most consequential temperature decision on a site, because it affects three things at once. The higher the permitted supply temperature, the more of the year the site runs on free cooling, the more valuable the recovered heat, and the less mechanical cooling is required.

The upper limit comes from the hardware manufacturer’s requirements, however, not from the designer’s preference. The class is therefore agreed with the hardware supplier at the start of the project. The heat exchanger approach belongs to the same question: the server loop supply temperature cannot be lower than the facility loop supply temperature, and a few degrees sit between them. If the hardware requires 32 °C supply and the approach is three degrees, the facility loop has to deliver water at no more than 29 °C.

Redundancy in both loops

Redundancy has to be designed loop by loop, and this is where a common error of reasoning occurs. Pump redundancy inside a CDU protects against a pump failure, but not against a failure of the unit. If one CDU serves several rack rows, its failure stops all of them regardless of how many pumps sit inside it.

The questions to resolve are:

• What may a single fault stop, and for how long?

• What are the N+1 or 2N requirements in each loop?

• Can each CDU or loop section be isolated for maintenance?

• Does capacity hold with one dry cooler out of service in summer?

• Do controls and power supplies withstand the same fault as the mechanical plant?

The last of these is the one most often left unchecked. A mechanically redundant system is not redundant if both pumps draw power from the same board or take their control from the same controller.

Isolation valves deserve their own attention. Serviceability in operation comes down to what can be shut off without stopping the hall, and that is decided in the pipework design. An isolation valve added afterwards requires draining the loop.

Scope splits

A scope split is the point where one supplier’s responsibility ends and another’s begins. A liquid-cooled hall has an unusual number of them, because the same loop involves the hardware manufacturer, the CDU supplier, the piping contractor and the facility contract.

The typical division is that the hardware manufacturer supplies the cold plates, rack manifolds and hoses, the CDU supplier the unit, the piping contractor the hall pipework, and the facility contract the heat rejection. The boundaries are not fixed, however, which is exactly why they have to be written down.

In practice these questions are worth resolving before the tender documents go out:

• Define the physical boundaries and the supplier responsible at each connection.

• Agree who specifies, supplies, fills and verifies the coolant.

• Assign flushing, air removal, balancing and flow verification.

• Agree who accepts each commissioning stage and how fluid or contamination issues affect warranty.

Flushing and cleanliness verification are the most critical of these, because neglecting them damages another supplier’s equipment. The chemistry requirements are covered in the article on liquid cooling water chemistry, filtration and material compatibility, and the commissioning stages in data center cooling: commissioning and handover.

Common problems

The temperature class is decided without the hardware supplier. Design proceeds on an assumption, and the actual requirement arrives once the pipework is already sized.

Redundancy is designed for the mechanics only. Power supply and automation remain a common point of failure.

Isolation valves are cut to save money. The saving is small and is repaid at the first maintenance job.

Scope splits are agreed verbally. They hold until something breaks.

Responsibility for cleanliness is never assigned. Everyone assumes someone else flushed the loop.

Finkova’s scope and next step

Finkova installs and commissions data center cooling in the Nordics, with instrumentation and lifecycle support. The work scope, OEM requirements and acceptance criteria are agreed for each project.

Contact Finkova with the project inputs above to discuss the relevant installation, testing and handover interfaces.