Article Featured in Data Centre Review
By: Gordon Johnson, Senior CFD Manager, Subzero Engineering
The current and growing industry focus on sustainability has highlighted the need to work towards energy efficient data center designs, which, in turn, should provide the perfect combination of sustainable operating conditions. The end result being truly optimized data center performance.
Computational Fluid Dynamics (CFD) plays a vital role when it comes to data center construction, retrofit, and helping improve the efficiency and performance of legacy data centers. It allows the owner or operator to simulate endless combinations of racks, cabinets, cabling, and mechanical and engineering (M&E) equipment for greater optimization.
The owners and operators of these facilities may not be aware that they are literally blowing a lot of (unnecessary) cold air into the aisles in order to keep the servers, switches, routers, and the storage units cool. What they will know, however, is that this wasted air is wasted money. Worse still, emissions will be increased, hot spots will remain an issue, and they will eventually experience equipment failures.
Many data center managers know that something has to change to overcome this unsustainable situation, but they may not have a plan or strategy. This is where CFD is critical.
CFD is essentially an MRI of the data center, and when it comes to informed decision-making, the mantra ‘if you can’t measure it, you can’t manage it’ still holds strong. However, in the case of CFD, ‘if you can’t measure it, you can’t improve it’ may be more appropriate.
What is a CFD analysis?
CFD software creates a 3D model of the physical attributes within a data center, as well as the location and performance characteristics of the cooling units, IT equipment, power systems, and other significant components of the data center.
This ‘data center MRI’ provides a complete, detailed thermal understanding of the data center, providing insight relating to the physical layout of the space and equipment; the power equipment thermal dissipation and rack power density; environmental conditions; cooling capacity and air distribution effectiveness. This is the baseline model or the “as is” snapshot of how the data center is currently operating.
From the information obtained, additional CFD modeling will recommend how to optimize the operating environment while reducing energy and maximizing efficiency.
Many key benefits of CFD analysis include assessment of the data center layout and its performance; validation of proposed design changes, considerations for efficiency and reliability; accurate prediction/simulation of the cooling design process; the ability to assess temperature and airflow modeling; and reconditions for a cost-effective and energy efficient design.
CFD provides clarity
A data center CFD analysis provides the visibility that is essential when it comes to understanding both your existing operating environment and any potential design changes. CFD can also be used to test out ‘what if’ scenarios when it comes to new data center construction or modernization projects.
In essence, a CFD provides detailed, data-driven analysis of the cold supply air and the hot exhaust air throughout the data center, and identifies the potential overheating of IT equipment “hot spots”. Such analysis highlights the way in which energy costs can be reduced by optimizing and maximizing data center cooling. CFD is a tool used to virtually test legacy (and new) data center environments for active temperature, pressure and airflow, and to optimize the mechanical system directly to server heat loads (kW).
Armed with data from the CFD analysis, a suitably experienced technology partner, such as Subzero Engineering, can advise customers on impactful data center design changes, upgrades, or tweaks to improve operating improvements – all of which will help optimize and reduce energy usage.
For example, CFD modeling can demonstrate what impact a cold aisle containment (CAC) system will have versus a hot aisle containment (HAC) system. It will advise which method will save more energy, full or partial containment. After containment is installed, it will show if computer room air conditioning (CRAC) units can be turned off and how low fan speeds can be reduced on computer room air handlers (CRAHs). Finally, it will also inform the user on how high the operating temperature can be increased, while following the recommended thermal guidelines from AHSRAE.
The end result is that CFD based design improvements will provide customers with significant PUE, WUE, KW, kWh, carbon, and water reductions.
A logical conclusion
One of the major benefits of using CFD modeling and analysis is that it can quickly demonstrate how a containment solution will immediately reduce a data center’s energy cost and lower its environmental footprint.
Regardless of which type of containment is used, large energy savings can be achieved by optimizing the airflow and raising the supply temperature, while simultaneously eliminating hotspots. Additionally, containment improves the benefits and economic feasibility of other efficiency measures, such as digital scroll compressors, electronic commutation (EC) and variable speed drives, data center infrastructure management (DCIM) software, and building controls.
Further, by separating the cold supply air from the hot exhaust air, containment saves energy, thus reducing operating costs. Yes, there is the initial investment cost of installing the containment solution, but the potential return on investment (ROI) of 10-14 months means that, very quickly, end-users will achieve payback. What’s more, from that point onwards, they’ll be saving energy, reducing operating expenditure (OpEX) money, and reducing carbon emissions to become more sustainable.
A savings summary
Since 2015, we’ve kept track of the various savings we have achieved for our customers through the careful, considered optimization of their data center environments. The numbers are impressive: our customers have saved just under 3.5 billion Kilowatt hours, reduced water consumption by one and a half billion gallons and have a total carbon reduction of over 3.2 million tons – the equivalent of planting 17 million trees.
Along the way, we’ve also been able to help customers achieve up to a 29% energy cost reduction, and been able to reduce PUE (Power Usage Effectiveness) by an average of 0.4 and WUE (Water Usage Effectiveness) by an average of 0.3.
Subzero Engineering believes that a data-driven approach to digital infrastructure is essential and thanks to the use of CFD, we can simultaneously help our customers increase data center density, operating efficiency, and environmental sustainability by implementing optimized containment solutions.
However you choose to address the challenge and opportunity of improving the efficiency of your data center, it’s essential that CFD is used as a key tool in this process. Without CFD, whether you choose to implement an optimization program yourself, or to engage with a technology partner, you’ll neither be able to understand how your data center is performing right now, nor what improvements can be made into the future.
By Sam Prudhomme, Vice President of Sales & Marketing
Recently, I was interviewed by a Computer Weekly journalist, Fleur Doidge, who was writing an article focusing on the quick wins when it comes to improving systems performance in the data center. Inspired by our discussion, I’ve come up with Subzero’s top 5 ways to boost data center utilization.
The data center industry, quite rightly, has an ongoing, major focus on how it can improve both the performance and energy efficiency of its facilities. That’s partly down to the perception that our industry are major consumers of energy and contribute a high volume of carbon emissions each year.
According to an article published in Science Magazine, data centers account for around 1% of global energy use. It’s clear that we need to improve our environmental performance and to ensure we never forget we’re part of the sustainability solution, but we should also remember that data center performance and energy efficiency improvements make great business sense.
While there are many, many issues to consider as part of a comprehensive, long-term strategy to both improve data center performance as well as to achieve carbon neutral status, this article focuses on the ‘low hanging fruit’ – relatively simple actions, which will have an immediate positive impact on your facility, and with an ROI measured in months rather than years.
Step 1 – As Easy as (free) CFD
Those of you who know Subzero Engineering well will not be surprised that Step 1 involves a Environmental Impact Evaluation (CFD) of your data center. We believe it all starts with the data and offer this service for free. It is a simple, efficient, and super-fast way of discovering just how your data center is performing right now – where the power is, where the heat is, and isn’t, hence where the cold air does or doesn’t need to be.
Step 2 – Using the Data
Once the Environmental Impact Evaluation (CFD) has been carried out, you’ll be armed with a large quantity of data about how your data center is performing. It’s highly likely that you’ll be presented with some really quick wins. For example, you’ll discover where the hotspots (points of efficiency leakage) are; and part of the solution may be something as simple as installing any necessary blanking panels.
Then again, the CFD data may highlight that Row 5 in Rack 6 is running 15 degrees hotter than anywhere else in the data center. You’ll be able to decide whether you need to move this stack to a better location where more cooling is available, or maybe you just need to open up the grate to optimize or increase the airflow.
Step 3 – The 3 ms: Measuring, monitoring & Modulation
A data center is a live environment. So, although the CFD analysis can identify and help to resolve what we might call any ‘permanent’ power and cooling issues, it’s essential that you monitor and measure the performance of the power and cooling plant in real-time. This is because data center variables such as the IT load and operating temperatures are in constant flux. With the right system you are able to modulate the airflow accordingly. For example, if the cooling needs to react to the load inside each rack and cabinet, as well as respond to the impact of, say, an extremely hot outside temperature.
Rather than blast a load of cold air into the data center and ‘hope’ that it keeps the IT hardware within operating tolerances, with the right monitoring solution, you can be confident that you can modulate the cold air as required right down to the rack level. This ensures that the cooling usage is as effective and energy efficient as possible.
Step 4 – Contain Your Excitement
How would you like to reduce your PUE by 0.4? Or to achieve a 29% reduction in data center energy consumption? Well, these are the average savings we achieve for our customers when they deploy one of our containment solutions.
The initial Environmental Impact Evaluation (CFD) we carry out also proves how this can be achieved – it compares and contrasts hot vs cold aisle containment and containment vs no containment. Furthermore, a containment solution ensures that Steps 1-3 really do achieve the maximum performance and energy efficiency improvements within the data center.
Without containment, you’ll still have hotspots – separating hot and cold air will be hit and miss and far from being optimized.
With containment, you can bring down the power consumption to cooling ratio close to a 1:1 match in KW consumed – that’s how the energy consumption/utility bill reduction is achieved.
As for the PUE reduction? Well, that’s achieved by smarter, more efficient use of an optimized combination of chilled water and the air conditioning fans. The US Environmental Protection Agency (EPA) estimates that containment can reduce fan energy consumption by up to 25% and deliver 20% savings at the cold water chiller.1
One final containment benefit – we supplied a containment solution to a colo customer and by separating the hot and cold air in their facility, we helped them to not only eliminate hotspots, but also to increase rack density by an average of 14%.
Step 5 – Turn the Lights out
This breadth of knowledge brings me to my final data center performance and energy efficiency improvement: turn out the lights. By that, I actually mean remove anything incandescent and go with an LED retrofit kit within your existing tray system. And then automate the lighting system.
It may be a while before a true lights out data center becomes the norm, if ever, but that’s no excuse not to ensure that your lighting system is as energy efficient as possible. By using LEDs and only using the lights when needed you’ll improve your energy efficiency as well as your bank balance.
While Subzero Engineering’s major focus is data center consultancy, using CFD analysis and containment solutions, to help drive performance and efficiency improvements we can also help owners/operators with their critical power infrastructure, DCIM and other solutions as required.
Download the 5 Steps to Improving Data Center Performance & Energy Efficiency pdf here.
1Recalibrating global data center energy-use estimates – Science Magazine 2018
Andy Connor – Channel Director, EMEA, Subzero Engineering
In 2020, the Uptime Institute’s, Andy Lawrence stated, “The average power usage effectiveness (PUE) ratio for a data center is 1.58, only marginally better than 7 years ago.”
This revelation may come as a welcome shock, and while it might be overstating the situation to characterise data center energy usage as the Internets ‘dirty secret’, there’s little doubt that the reality of the sectors carbon impact has been masked by the many headlines which focus solely on its sustainability successes.
Colt Data Centre Services, for example, recently announced that its operations across Europe are now fully powered by 100% renewable energy, while many members of the U.S. hyperscale community are publicly revealing their latest renewable energy projects and initiatives. Carbon offsetting is another idea quickly embraced by end-users, vendors and operators of all shapes and sizes, and while all of these activities, in part, contribute to data center sustainability improvements, they do not directly address the issue of data center power consumption.
Data Center Power Usage
Today there are many estimates as to the amount of power that data centers across the globe consume on an annual basis. Energy Innovation estimates that, in 2018, data centers likely consumed 205 terrawatt-hours (TWh), which equates to 1% of total global electricity. However, the authors of a paper published in ‘Global Energy Interconnection’ in June 2020 state that ‘data centers will become the world’s largest users of energy consumption, with the ratio rising from 3% in 2017 to 4.5% in 2025’.
The data differs again in the January 2020 Uptime Institute Journal, which reports EU data center energy consumption figures of 130 TWh in 2017, alongside Greenpeace’s 2018 Chinese data center figure of 160 TWh, which makes for a combined total of 290 TWh for China and Europe alone!
The fact is that as data center capacity increases, so will energy usage. And while we may not agree on the exact numbers, few would argue about the direction of travel. So what can we do to change the trajectory, and how can we begin to pinpoint consistency within sustainable strategies?
A Change In Demand
Demands for digital transformation are a key factor behind data center energy consumption, but rather than overload you with a tsunami of data, I would urge you to take a short time out to consider just how essential information technology has become to almost every factor of everyday life.
Take a typical workday, how do you communicate with colleagues, what does your role entail? Then think about your plans for the weekend (lockdown not withstanding), have you thought about the films you’ll watch, the apps you might use, or your personal connection to a data center?
Now try and imagine a future that also includes artificial intelligence (AI), augmented (AR) and virtual reality (VR), what impacts will this digital consumption have on the data center industry, and what does it mean for sustainability?
Those are big questions, many of which people outside the sector won’t have considered, but the answer may lie within a recent data center industry initiative to show us the way forward. Recently 25 companies and 17 associations across Europe joined together to sign the Climate Neutral Data Centre Pact, with the objective of making data centers climate neutral by 2030.
There’s clearly a growing momentum behind sustainability, but the industry needs to move beyond the environmental easy wins of the past few years and to start to address the factors that really address efficiency and PUE ratings – those which have hardly changed in seven years.
There’s also the question of is PUE truly enough to measure our carbon impact, so with this in mind, where can we begin?
Beginning With The Data
Back in 2005, Subzero Engineering started life as a Computational Fluid Dynamics (CFD) consultancy. At the time, a large percentage of the industry were using raised floors and experiencing issues with leakages. Yet, with a simple to use and accurate software solution we were able to show customers how they could analyse their data center infrastructure and take steps to both improve efficiency and reduce their environmental impact.
Fast-forward sixteen years and that approach has stayed with us. Today we’re an engineering-led solutions provider that helps world-leading businesses achieve a lower carbon footprint, greater efficiency, reduced operating costs and exceptional performance – and it all starts with the data.
For example, by showing customers the hot and cold air influences within their data center and helping them to analyse, optimise and retrofit their facilities, we believe we can help them find the perfect balance between sustainability and performance.
The proof is in the outcomes and today we keep a live record of the annual energy savings we’ve achieved for our customers. To-date they include:
- Total savings: $332M
- Total kW savings: 356kW
- Total kWh savings: 3BN kWh
- H2O savings (gallons): 1.5BN
- CO2 reductions (tons): +3M
However, while these data points show some of the gains that can be made by focusing on sustainability, two questions remain; how do organizations become more energy conscious, and what are the next steps they can take to become more sustainable?
Defining Next Steps
At Subzero Engineering we believe that beginning with a data-driven CFD report is the first step, and offers data center operators insight into how to drive efficiencies across all areas of their facility. This is not limited to airflow; it includes the Uninterruptible Power Supply (UPS), Computer Room Air Conditioners (CRAC), racks, IT and cooling fan speeds.
A CFD analysis also shows them how they can achieve a higher rack density, more computing power and help increase the operating temperature to gain both a higher performing and more streamlined, efficient data center.
This information is invaluable, offering both a starting point and a medium for creating a strategy that balances performance and efficiency. It also offers a means of truly understanding what kind of return on investment (ROI) they can expect from improving sustainability, especially in terms of reductions in energy and water usage, and lower carbon emissions.
Today energy efficiency and sustainability objectives have become key drivers for owners and operators. Subzero has always been a sustainability-engineering organization; it just so happens that in recent years ‘sustainability’ has become a key talking point for the industry.
Coming back to the data, a paper authored by Anders S.G. Andrae once presented three possible scenarios for data center electricity usage (TWh) by 2030. The best-case figure is 1,137; the expected figure is 2,967; and the worst case is 7,933. As an industry we cannot let the latter become a reality.
In the absence of the grown-up sustainability conversation that needs to happen soon, where more businesses and consumers become fully aware of the environmental consequences of their digital footprint, I believe more and more pressure will force our industry to perform better.
Sustainability, however, begins with data-driven action, and a free CFD analysis is a perfect place to start.
To learn more about modernizing or driving data center sustainability with the help of Subzero Engineering, connect with a local technical expert here.
This blog was first published on Intelligent CIO Europe in March 2021. To read the original online, click here.
We interviewed Gordon Johnson who is a certified data center design professional, Data Center Energy Practitioner (DCEP), CFD and electrical engineer regarding the use of CFD’s and containment.
Gordon, what is the principle way CFD’s are used with regard to containment?
We use CFD’s to determine two basic data sets. The first is the baseline, or the current airflow pattern. This initial CFD model shows supply intake temperatures to each cabinet. This model also determines the effectiveness of each AC unit as it relates to airflow volume, return air temperature, delta T, and supply air temperature.
The second model is the proposed design of the CFD engineer who uses the information from the base model to enact airflow management best practices to separate supply from return airflow. Typically several models are created in order to adjust airflow volume, set point temperatures, and adjust individual aisle supply volume.
Gordon, Are there situations in which the CFD engineer does not recommend containment?
Not really, because the entire basis of airflow management is the full separation of supply and return airflow. Anytime these two airflows mix there is a loss of energy and consistent supply temperature to the IT thermal load.
We have seen CFD’s used by manufactures to prove product effectiveness. What are some ways CFD’s are made to exaggerate product effectiveness?
Exaggerations usually stem from the principle known as GIGO, short for Garbage In, Garbage Out. This refers to the fact that computers operate by logical processes, and thus will unquestioningly process unintended, even nonsensical input data (garbage in) and produce undesired, often nonsensical output (garbage out).
Let me give you an example. Recently I recreated a CFD model that was used to explain the effectiveness of airflow deflectors. The purpose of the CFD was to show the energy savings difference between airflow deflectors and full containment. We found that certain key data points were inserted into the models that do not reflect industry standards. Key settings were adjusted to fully optimize energy savings without regard to potential changes to the environment. Any potentially adverse effects to the cooling system’s ability to maintain acceptable thermal parameters, due to environmental changes, are not revealed in the CFD model. Thus, the model was operating on a fine line that could not be adjusted without a significant impact on its ability to cool the IT load.
Can you give us any specifics?
The airflow volume was manually changed from 1 kW at 154 CFM to 1 kW at 120 CFM. Industry standard airflow is 154 CFM. The formula most commonly used is as such:
120 CFM airflow does not give the cooling system any margin for potential changes to the environment.
Another key area of unrealistic design is the placement of cabinet thermal load and high volume grates. The base model places high kW loads in specific, isolated areas surrounded by high volume grates. What then happens, if additional load is placed in areas of low volume airflow? Any changes to the rack kW in areas without high volume grates could not be accounted for. At the end of the day, any changes to the IT load would require an additional airflow management audit to determine what changes would affect the cooling solution. Thus, the proposed model is unrealistic because no data center would propose a cooling solution that would require regular modifications.
Are you recommending a CFD study every time you make changes to the data center thermal load?
No. a full separation supply and return airflow eliminates the guesswork with regards to the effect of air mixture. It also eliminates the need of specific high volume perforated tiles or grates to be placed in front of high kW loads. Instead, a CFD model would incorporate expected increases to the aisle thermal load. This falls in line with the “plus 1” kind of approach to cooling. Creating a positive pressure of supply air has many additional benefits, such as lowering IT equipment fan speed, and ensuring consistent supply temperature across the face of the IT intake.
Data centers should not be operated with little margin for changes or adjustments to the thermal load. That is why I always recommend a full containment solution with as close to 0% leakage as possible. This is always the most efficient way to run a data center, and always yields the best return on investment. The full containment solution, with no openings at the aisle-end doors or above the cabinets, will easily allow the contained cold aisles to operate with a slightly greater supply of air than is demanded. This in turn ensures that the cabinets in the fully contained aisle have a minimum temperature change from the bottom to the top of the rack, which allows the data center operator to easily choose predictable and reliable supply temperature set points for the cooling units. The result? Large energy savings, lower mean time between failures, and a more reliable data center.
What do you recommend as to the use of CFD studies and containment?
It’s important to create both an accurate baseline and a sustainable cooling solution design. This model will give data center operators a basis for an accurate representation of how things are being cooled. The proposed cooling solution can be used in numerous ways:
- Accurate energy savings
- Safe set point standards
- Future cabinet population predictions
- The ability to cool future kW increases
- Identify and eliminate potential hot spots
Subzero Engineering endorses accurate and realistic CFD modeling that considers real world situations in order to create real world solutions.