Google My Data Center      top of page
By Brian T. Soucy, P.E.

Perhaps I should clarify before you punch in a description of your facility into a popular search engine.  While the search results may be eye opening, what this article’s title is referring to is a state-of-the-art data center design concept. 

Internet companies such as Google have been making waves in the data center industry for over 5 years now.  The rash of internet data center construction brought about the PUE wars where one internet company after the other touted extremely high efficiencies (i.e. low PUEs).  Methods and designs used to achieve these performance ratings were closely guarded secrets and sources of much speculation until Facebook’s Open Compute Project (http://opencompute.org/).  There it was, laid out for the world to see, one road map followed to build high efficiency computing.  When the Open Compute story broke a short while ago it created considerable buzz in the industry and continued to be a popular topic of conversation at June’s 7x24 Exchange conference. 

The internet companies deserve a great deal of credit for driving innovation and asking the industry to consider alternatives.  Their hyper-efficient data centers will have a profound impact on their bottom line.  No different than a brick and mortar factory optimizing it’s supply chain or factory floor, these data factories strive to lower their overhead required to…give peopel directions and reviews for the nearest Thai food restaurant. 

So it begs the question, “Can I apply the same techniques used by the internet companies in my data center?”  The intent here is not to dismiss one idea or another but instead put concepts in the proper context and help us all better understand what a PUE in the neighborhood of 1.16 means to your enterprise data center.  To help frame the discussion let’s review these facilities in terms of the following three areas: cooling, power, IT equipment/operations.

Cooling
Global companies like a Google may have more options for geographical location, and they can choose climates where its possible to maximize the number of free cooling hours.   This is why locations such as the Pacific Northwest and the Netherlands and Denmark have been active markets.  Free cooling accounts for a substantial savings. Facebook, Yahoo and others use air-side free cooling where outdoor air is ducted directly into the computer rooms for cooling.  Using fan horsepower and adiabatic cooling these facilities can satisfy their cooling requirements most of the time without compressors, refrigeration or chilled water. While this is an option for new construction and some existing data centers, we've seen a certain amount of trepidation in bringing fresh air into the data center. Having said that, we have seen certain test labs that operated with outdoor air for ten years without incident and internet mega-centers are fully committed to it. To be ultra green this type of free cooling is a must but may not be right for your business.  In order to take the full benefit of free cooling you, as a data center operator, must be comfortable with the risks associated with outdoor air and have the luxury of sighting your data center in a climate that maximizes the opportunity for free cooling.

In addition to a favorable climate, these data centers increase the available free cooling hours by raising the allowable server inlet or cold aisle temperature.   Allowing 80-85 degree F temperature air at the server inlet as opposed to 70-75 degree F can delay the need to provide refrigeration and further lower the PUE.  In some cases, environments are controlled outside the ASHRAE TC9.9 Thermal Guidelines.  In order to drive down your PUE even further you, as a data center operator, must be comfortable with higher inlet temperatures to maximize the free cooling hours in any given year.  With today’s high density servers this may leave little margin for error and keep in mind that the hottest spot in your data center must be satisfied.  The internet company data centers are purposely built for these conditions.  Your facility may not be or may only allow only a marginal increase in inlet temperatures for a incremental benefit.  

Power
Internet data centers have also re-thought how they would provide reliable power.  For starters, in many cases they have eliminated a centralized UPS system or systems in favor of a battery back up at the server level.  The advantages of this approach include an overall reduction in capital cost and a reduction in losses associated with converting power.  They no longer have to take street power, convert it to DC, convert it back to AC, transform it and deliver it to the IT equipment, where it is converted back to DC.  Instead they will take street power direct to the servers, convert to DC, provide battery back-up at that point of connection and in turn provide that DC to the servers.  A disadvantage of this approach is reliance on many individual batteries distributed throughout the data center as opposed to a central, redundant plant.  This could present unique challenges for maintenance.  Also, servers are now dependent on the performance of its local batteries.  Failures will be localized but the increase component count could increase the probability of local server failures.

As mentioned above, the main power distribution system does not include central UPS systems and limits the number of transformations between the street and the servers.  In many cases capacity is more fully utilized than a traditional 2N data center.  With the ability to roll out thousands of servers at a time using containerized deployments the internet data centers can better match the capacity with demand. 

IT Equipment
Companies like Google and Facebook go to great lengths to build custom application specific servers.  These servers contain specialized power supplies and are stripped of unnecessary components that would otherwise needlessly waste energy.  The server power supplies are fitted with local UPSs or designed to accept non-standard input voltage to improve efficiency.  In some cases their primary source of power is generator backed street power with a secondary source backed up by battery.  These companies will deploy thousands of servers at a time; all configured the same, all performing the same operations.  Large deployments of identical servers allow them to take advantage of economies of scale and invest in customization. 

Another benefit of having many servers performing the same function, or sharing the work, is that no server is more important than another.  A CNET report indicated that Google would deploy over 1,800 machines in a cluster and in their typical first year of operation 500 to 1,000 machines would fail due to a failed power distribution unit and there is a 50% chance the cluster would overheat taking down most servers and requiring 1 to 2 days to restore.  This type of performance would be unacceptable in an enterprise data center but a reality when pushing the envelope. 

So what gives?  How are companies like Google and Facebook able to live with these design characteristics and tolerate the incumbent shortcomings?  It’s simple really; their data center operations and availability requirements are fundamentally different than a typical enterprise data center.  Not to say these data centers do not require continuous uptime and availability.  However, their applications are designed to work around the occasional failure of one or many servers.  They can shift and share work across their environment so failures stay local and transparent to their end-users.  For them their designs do not have shortcomings at all but instead employ practical engineering principles aligned with their operational and availability requirements.  They’ve optimized just about every component and sub-system in the chain to take advantage of a unique data center operational environment.  It would be very difficult for most institutions and enterprise data centers to put in practice all of these design methodologies today but the internet companies’ innovation has certainly received a good deal of attention.  Perhaps with greater penetration of cloud computing the future of data centers just might look more like Google data centers today.

top of page

Computer Room Air Conditioners or CRACs have been the default choice for years as the cooling unit used for data centers however that standing has been challenged with a host of new products hitting the market over the past few years as well as the reintroduction of traditional air handling equipment.  These new equipment offerings were spurred by problems created from higher densities on the raised floor to address a need for more cooling capacity and the problems created by exhaust air recirculating back to the rack inlet.  In this article we are going to focus on the options available for selecting your terminal cooling units.

The most common commercially available equipment for data centers will fall into one of three categories; CRACs, In-Row coolers or Built up Air Handling Units.  At one end of the spectrum is a variety of offerings of In-Row Coolers and as their name implies, they are placed in-row right next to the rack. On the other end is the Built-up Air Handling Unit that can be located farther away from the load and provide as much incremental capacity as you choose. In the middle is the familiar computer room air conditioner/handler (CRAC or CRAH). We’ll discuss some pros and cons of each to help you evaluate which configuration works best for your application.  Any one of the three concepts can be made to work for any data center; the key is finding the best match for your needs.  At the start of each new project, consider the following questions:

• Should you separate the data center room from the cooling equipment room?
• Should you eliminate chilled water or refrigeration piping from your data center floor?
• Should you eliminate the raised floor?
• Can you use containment walls, chimneys or curtains to improve efficiency?

First let’s discuss In-Row Coolers which are smaller by design in both size and capacity than traditional CRACs because the intent is to put them as close to the load as possible.  In-Row Coolers do not need to push the air very far because they are close to the server exhaust and supply cool air very close to the rack inlets.  Although the raised floor can be eliminated with In-Row Coolers, that choice limits the piping for each cooler to be overhead and sets up competition for space with the electrical power, data cable, fire protection and lighting.  Despite assurances from manufactures that overhead liquid leaks are not a problem, the possibility adds to the level of risk.  Under floor piping greatly reduces this exposure but adds the cost of a raised floor.  Since these are the smallest incremental capacity units available you will need a higher ratio of In-Row coolers per rack to serve the load, which will increase the number of piping connections, power wiring terminations, controls points, and serviceable components to maintain.  This installation and service work occurs on the data center floor in close proximity to your racks.  In-Rows allow you to match your cooling capacity more closely to your load growth however this configuration may elevate the dollars per ton of installed cooling equipment and may not be the best alternative for large scale projects. Control of In-Row Coolers is limited to the manufacture’s packaged offering and interconnection to a Building Automation System (BAS) may offer limited functionality at the BAS user interface.  One of the best advantages of an In-Row Cooler is their compact footprint.  This makes them well suited as supplemental coolers for rooms that have outgrown existing cooling capacity or smaller data centers with limited floor space or headroom for cooling equipment.  As is true with any system, In-Rows will perform better with containment although careful consideration must be given to the controls set up.   A facility owner or manger needs to answer the key questions above to decide if In-Rows are right for the project.

On the opposite end of the spectrum is the Built-up Air Handling Unit (AHU).  As the name implies you can build these units any way you like.  This requires some level of understanding about the options beyond the simple matchup between your kW load and a CRACs ability to deliver that amount of cooling.   AHU customization options include individual selection of the fans, coils, filters, humidification, reheat and controls.  Fans are selected for both CFM airflow rates as well as pressure output which is key to having the ability to move the cooling unit out of the data center room. Cooling coil selection is also at your control allowing you to pick a coil with better heat transfer capability which is a key for improving overall system efficiency.  With custom coil selection capability you gain control of whether you want to pay a higher first cost for a larger copper and aluminum coil surface area or have a smaller coil with lower first cost resulting in a efficiency penalty over the life of the equipment.  These coil decisions are made for you with typical packaged equipment, sometimes with the goal of offering you an attractive first cost.  A humidifier section can be added that allows you to use wetted evaporative media and waste heat from your data center to evaporate water.  Traditional manifold humidifiers may be used with electric or other heat sources like fossil fuel boilers.    A filtration section with prefilter and final filter can accommodate systems that use outdoor air for cooling that needs greater filtration.  Built-up Air Handlers have the option to be configured for economizer cooling using outdoor air with a separate return air fan that is used to manage the positive pressure in the space.  Other forms of indirect free cooling using air to air heat exchangers are also possible using this platform.  Controls for an AHU are completely flexible and are tailored to your needs.  AHU controls interface seamlessly with the rest of your Building Automation System when you use controllers by your BAS provider.  Designing and building your controls from the ground up is a process that involves more planning and sometimes more cost then data center managers are accustom to since the familiar CRACs often come with simplistic package controls.  Custom air handler controls are common outside the data center industry and are used for typical office buildings, schools, and retail.   Another advantage of AHU over other terminal units is your ability to decide how many incremental cooling units you will have to serve a space.  You can take the total cooling load for the room, divide it by 2, 3 or 4, select the appropriate unit capacity and then add units for N+1 , N+2 or N+N redundancy.  This can greatly limit the number of units that need to be piped, wired, controlled and serviced.  For example, a cooling demand that may require 20 or more CRAC units could be satisfied with 4 AHUs.  The system lends itself well to a contained room with either overhead ducting or raised floor air distribution.   A major disadvantage of Built-up Air Handling Units is the additional space and physical size needed to house the unit.  The data center industry has become accustom to the tidy footprint of a CRAC which packs a great deal of functionality in to a very compact box, this is not the case for the AHU. 

CRAC units still have their place in the industry and have improved in recent years with variable speed drives for capacity modulation and better connectivity to avoid fighting between units.  There is a great deal of existing data center real estate out there that is already fitted with a raised floor to accept downflow CRACs and is already equipped with infrastructure like under floor piping that may have years of useful life left.  For many of these existing spaces it may not be practical to use adjoining rooms to fit in a large custom air handler nor do the buildings have the headroom for ductwork or large AHUs.  For existing facilities already configured as a traditional raised floor, CRACs will often continue to be the most economical choice when refreshing an area with new technology.  CRACs must be in close proximity to the load they cool since by design their fan pressure is too low to distribute cooling an appreciable distance.  They can be made to work with containment and should have the controls reconfigured to respond to the discharge air or cold contained room temperature rather than the hot exhaust temperature in the return of the unit to achieve rack inlet temperature control.    CRACs can interface with an existing BAS to receive alarms, rotate for equal runtime, or perform lead lag control in the event of a unit failure although this kind of BAS interface can be more costly then controlling a smaller number of high capacity AHUs.  CRACs are simple to install and set up with packaged controls but this simplicity can become a hindrance if you want or need greater functionality.  

top of page

The design of data center electrical systems requires knowledge of the various operational requirements and code/client requirements. When reviewing the requirements of the National Electric Code (NEC), there are instances where complying causes concerns for the actual system installation or operation. For instance, the emergency power off (EPO) system has always been considered a compromising factor when considering overall system operational reliability for a data center. The NEC Article 645 defines the requirements for this system such as locating the disconnecting means at the principal exit(s) from the data center, clearly indentified and readily accessible.  Also, the method of operation had to be “pushing” the button when a push button type of switch is employed (other controls were permitted).  The intent is to make the EPO system as easy as possible to activate in an emergency situation both from a location and operational perspective. Designing a system that’s easy and convenient to activate on the one hand introduces an increased risk of unwanted activation on the other. This is an issue that is and has always existed for both the system designer and IT manager.

This code requirement has been present for over 40 years and the NEC 2011 edition has addressed this issue by modifying/clarifying requirements for the EPO system. The code panel for this article met with and discussed issues of concern with organizations that were affected by these requirements. Out of these discussions developed requirements that would better meld the intent of the article with the concerns of the designers and operations managers. With regards to the disconnecting means, it no longer must be located at the principle exists of the data room. It can be located at “approved locations readily accessible in case of fire to authorized personnel and emergency responders”. The reference to having a “push” motion when employing this type of switch has been eliminated. The modification/removal of these two requirements allow designers and IT managers to rest a little easier knowing that there are options to satisfy both ease of operation and system misuse. The determination of the disconnecting means location can be such that a mutually agreed area by both the local authority having jurisdiction and operational people will minimize unwanted access to and system activation. For instance, the location can be at a remote monitoring site that will respond to various security/ alarm events or in a data center location which will minimize accidental or unwanted system activation.

This change in the code requirements demonstrates the importance of developing regulations which consider both the code’s intent and effect on the people who use it. In the case of this particular situation, the change has increased the EPO system’s reliability.

top of page

As we are all well on our way to becoming more green, we need to consider what your PUE and that of other companies really means.  At this point, because we are still at such an early age of monitoring, you must keep in mind that all PUEs are not alike, and comparing one companies PUE to that of another is really not an exact comparison.  Tracking your PUE is very helpful, in determining your progress to conserve energy, but only as it pertains to your facility.  All companies are different and have their own requirements as well as means to track their own PUE, and that is why there has been the development of the different classifications of PUE.  There is an effort to someday compare PUEs, but as of today your PUE is your own personal gauge as to how your required infrastructure is becoming more efficient.

top of page