In the world of facility management, timely action and effective communication are crucial. This cautionary tale highlights the consequences of delayed assessments, grounding issues, and interdepartmental conflicts that led to costly outages and operational disruptions.
For over a year, this client’s building manager struggled to obtain approval for a facility assessment to confirm equipment operation and verify system redundancy. Management’s delay in uncovering system weaknesses was augmented by a costly failure of a utility substation transformer. It was only after this failure that the assessment was approved, along with a root cause analysis (RCA).
We were contacted when this client experienced a catastrophic failure of the 115kV primary transformer at its utility substation, which led to a utility outage and a subsequent failure of the customers’ redundant systems. We assessed the building’s electrical and mechanical systems, single points of failure, equipment condition, physical layout, operations and maintenance procedures, and other aspects of the facility that could impact production and long-term use.
We traced the failure of the customers’ redundant systems to grounding and bonding issues, along with improper server dual coding. Fault currents traveled through the building’s grounding system, damaging UPS controllers due to ungrounded racks and improper cable connections. At our recommendation, the client’s engineers split the UPS outputs, verified proper A and B cording, and addressed the grounding issues. The replacement utility transformer later experienced a similar fault without causing any issues at the customer site, confirming the effectiveness of these fixes. Had the customer had a proactive approach instead of delaying action, we could have helped reduce or eliminate the original outage caused by the utility substation transformer failure.
A Tale of Separate and Unequal Departments
While uncovering the cause of the failure, we discovered interdepartmental issues that contributed to the delay in securing the assessment. A lack of trust and poor communication between IT and facilities staff led to blame and animosity, which negatively impacted the workplace culture. The animosity was so severe that team members avoided each other in the hallways.
A systematic process of verifying power circuits and correcting problems was developed over several months, bringing the IT and facilities teams together. Their cooperative effort to solve the problem transformed their relationship into one of cooperation and friendship, ultimately improving team dynamics.
Both management and staff, including the building’s engineer, also lacked an understanding of grounding principles and the impact of nearby substations, which contributed to the issues that arose. Education during the RCA process helped improve understanding and prevent future problems.
Prevention is Key
In facility management, taking timely action and communicating effectively are essential. The delayed $30,000 project approval resulted in a $3–4 million outage cost. The financial implications served as a stark reminder of the high cost of inaction and the importance of timely maintenance and communication. By addressing these issues proactively, organizations can prevent costly failures, ensure operational resilience, and improve team dynamics.
Don’t wait for a failure to act; assess your facility’s electrical systems today. By doing so, you can prevent costly outages and improve overall operational efficiency. Contact our team of experts to schedule an assessment.