Power infrastructure never sleeps and that is why MCP operates 24/7/365.
Team MCP had a late night getting lit this hump day! A little humor from the tech corner, as we headed out to a data center to load test two parallel 750kVA UPS modules that had blown up in grand fashion last week. In this case, Mission Critical Power came in on the back end to resolve a problem created by a lack of maintenance on aged equipment. Unfortunately, the missed maintenance reviews led to a big boom, a stinky smell, and a few frantic site representatives.
No worries. MCP was on hand quickly to resolve the problem. The team completed a heavy repair/rebuild solution on the fly. Now we are back to finish the job with load bank testing as the final step before taking the load live. My bird’s-eye view onsite began today. I have a hybrid role within our organization and am not always in the field; however, I enjoy seeing the troubleshooting live and being a part of the entourage that gets to watch the “switches” get flipped from a safe distance. This is my lens represented in this post, as unique as it is.
First, I am glad to be able to learn from MCP’s Master Technician, Howard. It’s cool to see the banter of the engineers, the calmness of our experienced team, and then the sweat beads on the foreheads of the executive-level team that flew in for observation. Meanwhile, I am imagining myself watching a movie being filmed “live” while eating my popcorn in my seat. The stage was ironically set: As we entered the UPS room, the physical effects of the explosion were visible—some charring on the units that were rebuilt, but also the graveyard of discarded burnt parts and wires were present. A cool sight, but also a reminder of how important safety is in our day-to-day roles.
We began our work by first checking the safety bypasses and ensuring that the equipment would be safe to work with—of course, all while wearing proper PPE in case ARC flashing occurred. The helmet, the face shield, the massive rubber gloves, and the oversized safety jacket took me to a visual reminder of “The Hurt Locker.” With safety checks complete, we were able to move on to load testing. Lockout/tagout procedures were followed. The next stage of testing included a ramp-up of voltage. We had two load banks running, and we started at a 50kW load on each machine to give us a test level of 100kW. The process included waiting time for each step to ensure that the load was properly being carried and that the two systems were able to communicate and share the load without the risk of another incident. The machines were in sync, and no explosive divorce was looming, per the analogy I heard in the whispers! Techs can be funny even during high intensity situations.
Power it up! The load was increased on each load bank by 50kW to allow us to step up to 200kW until we reached our goal of a 700kW load test. We continually checked the load sharing on the machines to ensure an acceptable deviation between the two units. As we wrapped up, we then allowed the system to run for a 30-minute test cycle at 700kW. The test went smoothly with no safety concerns. The warmth of the load banks outside was a nice little space heater of sorts to give you an idea of the power we were using. Once we finished, we went into the “everybody cleans up, clean up” state (cue the Barney song). It was time for us to slowly begin stepping down the loads on the machines. The site equipment was end of life causing us to step the load banks down in increments of 50kW per machine until we reached our starting value of 0 on each. At zero we had fully disconnected the load; we ensured that the customer’s system was running smoothly with no issues. After we cleared the load banks and their required cabling, we then cleared the room for safety to ensure that when we flipped the customer’s breakers to allow the system to be turned back online, we had a few degrees of separation in the worst-case scenario. The system was restarted without any issues. Another job well done for Team MCP and the rest of the crew on hand. This job ended up having us back on the home front near midnight, proving how power never sleeps. One thing we pride ourselves on is being a 24/7/365 company that can respond to any customer issues, literally any day and any hour.
If you have questions about load bank testing, UPS maintenance, or emergency services, don’t hesitate to give me a call. Thanks for following along in a DIML at MCP!
-Chris Collins, MCP Operations Team
