Chris Brown, Ryan Orr and Ed Rafter, Uptime Institute
Many data center owners and others commonly believe that commissioning takes place only in the last few days before the facility enters into operation. In reality, data center commissioning is a continuous process that, when executed properly, helps ensure that the systems will meet mission critical objectives, design intent, and contract documents. The commissioning process should begin at project inception and continue through the life of the data center.
Uptime Institute’s extensive global field experience reveals that many of the problems, and subsequent consequences, observed in operational facilities could have been identified and remediated during a thorough commissioning process. Rigorous, comprehensive commissioning reduces initial failure rates, ensures that the data center functions as designed, and verifies facility operations capabilities—setting up Operations for success. At the outset of the commissioning program development, the owner and commissioning agent (CxA) should identify the important elements and benchmarks for each phase of the data center life cycle. Each element and benchmark must be executed successfully during commissioning to ensure the data center is rigorously examined prior to operations.
Commissioning tests some of the most important operations a data center will perform over its life and helps ease the transition between site development and daily operations.
- Verifies that the equipment and systems operate as designed by the Engineer-of-Record
- Provides a baseline for how the facility should perform throughout the rest of its life
- Affords the best opportunity for Operations to become familiar with how systems operate and test and verify operational procedures without risking critical IT loads
- Determines the performance limits of a data center—the most overlooked benefit of commissioning
Selecting a Commissioning Agent
Selecting the CxA in the Pre-Design phase allows the CxA to help develop the commissioning program, the budget, and schedule.
The CxA should have and provide:
- Appropriate staff to support the technical requirements of the project
- Experience with mission critical/data center facility commissioning
- Experience with the project’s known topologies and technologies
- Sample commissioning documents (e.g., Commissioning Plan, Method Statements, Commissioning Scripts, System Manual)
- Commissioning certifications
- Client referrals
The CxA should be an independent third party, contracted directly to the owner to ensure the owner’s interests are held primary.
Ideally, the CxA should not be an employee of the construction contractors or architect/engineering firms. When a third-party CxA is not a viable choice, the best alternative would be a representative from the owner’s team when the technical expertise is available within the company. When the owner’s team does not have the technical expertise required, a third-party mechanical or electrical contractor with commissioning experience could be utilized. Of course, cost is a factor in selecting a CxA, but it should not be allowed to compromise quality and rigor.
The decision as to whether or not the design intent is to be scalable or use shared infrastructure will have a large effect on subsequent implementation and commissioning phases. Shared infrastructure systems in an incremental buildout can potentially increase the risk associated with future commissioning phases. Careful planning can mitigate this risk. Where shared infrastructure is to be used in a phased implementation, the OPR or BOD should highlight the importance of including design features that will allow a full and rigorous commissioning process at each phase of project implementation.
Re-Commissioning Future Phases
Commissioning should be performed any time new infrastructure is installed or any time there is a significant change to the configuration of existing infrastructure. This could include planned expansion of the data center or major replacements.
In data centers that are built to be scalable, it is imperative that commissioning be just as rigorous for the follow-on infrastructure deployments to minimize risk to the facility. Commissioning activities undoubtedly add risk to the data center, especially where infrastructure systems are shared. However, this risk must be weighed against the risk of performing the associated commissioning tests. If a component or system is not going to perform as expected, the owner must decide if it is better to have this occur during a planned commissioning activity or during an unplanned failure. While performing a rigorous commissioning program during the initial buildout may prove the concept of the design, the facility could potentially be at risk if all of the new infrastructure components and systems are not tested rigorously.
These types of commissioning activities, by their very nature, occur while the systems are supporting critical IT load. In these instances, the operations team best knows how these activities may impact that mission critical load. During re-commissioning or incremental commissioning, the operations team should be working in very close collaboration with the commissioning team to ensure the integrity of the data center. Additionally, if re-commissioning involves changes to the configuration of the data center, Operations needs full awareness so that operating procedures that impact maintenance and emergency activities can be updated and tested completely.
The best way to mitigate the risk of re-commissioning efforts or for follow-on phases is to ensure that the facility is properly and extensively commissioned when it is originally built. And, as part of a rigorous re-commissioning program, all of the points discussed in this paper for standard commissioning apply to the re-commissioning efforts. However, due to the higher level of risk with these activities, there are some additional requirements.
Re-Commissioning and Future Installation Phase Technical Requirements
All of the technical requirements previously provided apply to re-commissioning and future installation phases. However, follow-up commissioning activities also require the following special considerations by the CxA:
- Adequate notice must be provided to service owners about the schedule, duration, risk, and countermeasures in place for the re-commissioning activities in order to gain concurrence from IT end users.
- For facilities that are based on a dual-corded IT equipment topology, the owner and Operations should verify that the existing critical load is appropriately dual corded where systems that support installed IT loads are to be commissioned.
- As load banks can introduce contaminates, load bank placement should be considered carefully so as not to impact the existing critical IT equipment.
- Detailed commissioning scripts must be prepared and followed during commissioning to ensure minimal risk to existing IT equipment. Priority should be given to the live production IT environment, and back-out procedures should be in place to ensure an optimal mean time to recovery (MTTR) in case of a power down event.
- Seasonal testing of the systems should be performed to verify performance in a variety of climatic conditions, including extreme ambient conditions. This also ensures that economizers, where used, will be tested properly.
Commissioning activities represent a unique opportunity for data center owners. The ability to rigorously test the capabilities of the critical infrastructure that support the data center without any risk to mission critical IT loads is an opportunity that should be capitalized on to the maximum possible extent. Uptime Institute observes that this critical opportunity is being wasted far too often in data center facilities, with not nearly enough emphasis on the rigor and depth of the commissioning program required for a mission critical facility until critical IT hardware is already connected.
A well-planned and executed commissioning program will help validate the capital investment in the facility to date. It will also put the operations team in a far better position to manage and operate the critical infrastructure for the rest of the data center’s useful life, and ultimately ensure that the facility realizes its full potential.
For a full explanation of data center commissioning, please see The Uptime Institute Journal.