The night my computers died…

And how I became intimate with my water heater.

Suddenly, all the computers in my office fell silent but strangely the room lights were still on. After initial panic and bewilderment, I was able to solve the mystery and was again reminded of the value of standards and budgeting to do it right from the beginning (pay now to avoid paying more later).

Even though I did not immediately respond by formally establishing a structured problem solving methodology with an eight step discipline (or other QMS variant), my engineering background intuitively guided me through a similar process. This was also a reminder that troubleshooting very simple systems without advanced preparation can be fairly complex and time consuming, therefore proactive preparation for complex systems is essential.

The good news was that I was working on my laptop and even though the external monitor was now off, the laptop was up running on batteries and I was able to save my work. By switching over to the internal display I was able to save the presentation and cleanly shutdown my laptop. As with “mission critical systems” found in data centers and production environments which have backup power systems, the potential value of the lost work not to mention the effort to restore or recover far outweighs the small cost of the a desktop UPS even if never used. I’ve just added a few to my shopping list (before buying any new office “toys”).

With the room lights still on, the power outage appeared to be localized to the outlets in my office. From a standardization perspective, the benefit of placing light fixtures and outlets on separate circuits became obvious: you can still see when you trip a circuit breaker or otherwise loose power to your outlets. I was certainly appreciative that someone previously figured this out and it become standard practice (possibly included in the electrical code), even though this adds to the cost of wiring a home. This is a reasonable cost to incur up front to avoid issues later.

We’ve been slowly building up our “institutional knowledge” in regards to the “fuse box“. Over time, we’ve mapped out many of the vaguely labeled circuit breakers as to what they control. Wouldn’t it be great if we all had instant access to an accurate wiring diagram or schematic of our house wiring? Our house was built in 1968 and has had at least three prior owners; surely things have changed from the original prints even if they were still available. And creating this documentation from scratch is unlikely to make anyone’s list of top priorities. I ended up paying more in troubleshooting effort since it took longer to determine which circuit powers the outlets versus if I had I had done this when the system was working. I am not obsessive enough to go and label all my outlets with the breaker number like you will see in a commercial building, but I’m considering it…

Since we did not find a tripped or sticky breaker, we used the process of elimination on the seven vaguely labeled “Gen Plug” breakers and found one that did not appear to switch any outlets. This might have been a good excuse to go buy another tool but both the hour and low ROI of a circuit tracer eliminated that course of action. Before my father and I (the team we assembled – Step D1) opened the panel to investigate the suspect breaker, we decide to Define the Problem (Step D2). And we used another piece of sage managerial advice to look at the problem again when rested in the morning. In doing so, we collected additional data that the outlets in both adjacent bathrooms were also not functioning.

Since the hall bath outlet was the closest to the panel we decided to open it up first. However, we found nothing amiss other than it wasn’t a GFI outlet. We promptly replaced it with a GFI since this is another case where it is better to pay now versus having a disaster later.

I thought we could determine which outlets were on the circuit by looking at how the wiring ran from the outlets to the panel. Had I remembered the adage “plumbers crawl, electricians climb” prior to crawling under the house and finding almost no wiring there, I would have headed directly for the attic instead. This convention has several practical aspects (keeping wires dry, coordinating activities on the jobsite, etc.) which makes it valuable.

Our attic has additional “blow in” insulation which is not only messy but covers all the wiring making it a challenge to work in the attic. Not finding how the circuit arrived at the bathrooms or my office made me suspicious that we had missed something. We did find the furnace and outlet in the utility closet were also controlled by the suspicious breaker – i.e. they did still have power on when the breaker was on. This caused us to doubt the circuit breaker itself was the root cause.

Back to standardization, by code all splices or connections of wiring need to be done inside an accessible junction box prohibiting a connection inside a wall. This too increases the cost of wiring a house, but as this situation demonstrated it makes it highly unlikely the problem is under the sheet rock where it would be the most difficult to find and repair. And it focused our problem solving (having skipped interim containment actions – Step D3) on the junction box in the utility closet with the outlet and furnace cut-off switch. Upon disassembly and inspection we found root cause (Step D4) where the wiring leading form the junction box to the downstream outlets had rusted and broken. Having found no signs of recent moisture, the rust was most likely due to either moisture entering the junction box from the hot water heater overflowing or being replaced in prior years.

The poor location of the junction box, hidden behind the water heater, was due either to incompetence (since you would need to reach around a malfunctioning furnace to turn it off) or each of the appliances having been replaced at least once (most likely with units larger than the original). As such, we spent several hours leaning against and hugging the water heater to replace the original junction box. (Step D5/D6: choose and implement corrective actions.) We relocated the new junction box with a new furnace cut-off switch and outlet, higher up the wall and out of the way of the heater and furnace. This is work that should have been done previously when the furnace or water heater was replaced to avoid the very high deferred cost.

As to preventing a recurrence (Step D7), I now know to watch plumbers very carefully to make sure my outlets stay dry! To finish our problem solving methodology (Step D8 -congratulate your team): a big thank you to my father who happened to be on hand to help out.

Reflecting on the challenges of this problem, it is easy to see the potential magnitude of the issues and how difficult it would be to solve and repair it if we did not have electrical standards (or if electricians or contractors failed to observe them). Without these standards, I would probably have big holes in my walls! Luckily, I was able to tackle a forty year old system without documentation and without a specialist. Thankfully, the system complexity was fairly low making this possible.

It also demonstrated that deferring work may cause you to pay more later than doing it right from the beginning. Consider this when eliminating features, ignoring standards or deferring implementation costs such as adequate documentation. Do everyone a favor and do it right since you don’t know who might be attempting to service your products in the future!