Planning for Failure

When I worked in the power industry everyone understood that the cost of a power outage was high, but it was impossible to put a precise value on it. There is the reputational damage, the specific costs of payouts to businesses and residences that are impacted (depending on your jurisdiction), the cost of restoring service (typically it is “all hands” in terms of available personnel and equipment), and finally the loss of trust by your all-important regulator when you come back later and ask for an inevitable price increase for your customers.

The other, more subtle, cost of outages is the fact that businesses and residents must plan for unreliable power sources, and invest in backup generation which includes fuel, testing, etc… I would call this “planning for failure”. Over time, this also causes businesses to consider exiting the grid entirely in one form or another when they are large and capable enough, causing the remaining fixed costs to be borne by the remaining customers.

Here in Portland right now we are dealing with a major outage, as a fire caused a power outage to over 2000 customers downtown near the Pearl district. This isn’t 2000 customers… most of these meters are large businesses and buildings and not individual houses. In practical terms, the downtown Target is probably closed, Powell’s bookstore (a major tourist attraction) is closed, and many, many other smaller businesses and restaurants. It would be similar to a power outage taking out most of River North in Chicago where I used to live.

Luckily I live in a building with a backup generator, and they have fuel for 3 days, so we likely will be unaffected. That’s what you get when you pay more for a recently built class A apartment rather than an older vintage walkup. But many, many folks are going to be impacted by this (it was over 90 degrees yesterday) and many restaurants are going to have to throw out their food on top of losing a couple of days’ worth of customers.

As we re-think electricity and the grid entirely it is important to consider reliability in the equation. I believe that many individuals and businesses just take power for granted until it isn’t there anymore. This challenge will likely be exacerbated by renewables and solar power… in this outage it is a distribution system failure, but intermittent generation of power is another variable in the reliability equation.

Cross posted at LITGM

20 thoughts on “Planning for Failure”

  1. There’s also the connection between the electrical grid and other types of infrastructure, including water, sewer, and gas pipelines.

    I’m not sure to what degree the typical water distribution system has backup generators…my perception is that many of them do not; these are large, power-hungry pumps and full backup would be expensive. I *think* I’ve read that backup is more common for sewer systems, but not sure of this.

    Also, much of the nat gas pipeline system uses pumps that are powered by turbines or engines running of the gas itself, but apparently there are still elements that are grid-dependent.

  2. I was in Boston November 1965 when the great northeast blackout occurred.

    I was in the morbidity/mortality conference in the Mass General hospital when the lights went out.

    The hospital had been convinced to disconnect their backup generator when the northeast grid was set up. It was still on the hospital grounds but could not be started or connected.

    My wife was at a friend’s house in Boston and was in the act of screwing in a light bulb when it happened. She was convinced she had blown their fuse until a short while later she was driving home and heard the report on the radio.

    It was just about five o’clock and the power did not come back until about midnight.

    The only place in Boston with lights was the Edison plant on the Charles River, which was brightly lit.

    The MGH had to do an appendectomy that evening and boiled instruments and used battery lights for the operation.

  3. When you moved to Portland you must’ve sent their weather back to Chicago. It’s been cold and rainy for weeks. I’m preparing to launch the ark any day now.

  4. It is not just the power grid. Many institutions and elements of our infrastructure and governance were developed over time by serious people who had experience with hardship and deprivation and such challenges as the Great War and WWII and the Depression. With increased affluence and separation from the realities of agrarian and early industrial life, there has been a loss of awareness of how badly things can go awry, and how important it is to build robustness and even redundancy into many of the structures and systems that we depend on for basic security and comforts.

    The moral and intellectual foundation of much of modern civilization has been eroded, and with it some skill sets and the determination to ensure continuation of public services in the face of expected and unexpected adversities. Individuals can make some compensating adjustments, but we are all at the mercy of our own complex systems to a degree that is becoming a matter of deep concern.

  5. Individuals can make some compensating adjustments, but we are all at the mercy of our own complex systems to a degree that is becoming a matter of deep concern.

    I’m uneasy about the cars, for one. I sometimes think I should get back the 1996 Nissan truck I gave my daughter. She still has it and it would probably be less susceptible to the EMP than what I have now.

    When I started looking at homes in Tucson, I was looking for houses on four or five acres, thinking about a propane tank and generator. We wound up with 1 acre and more convenience to shops and restaurants but I still worry about civil disorder. At least we got out of California.

  6. There are many articles about the problems with the Australian power systems reliability and how industries, especially the aluminum industry are cutting back their plants and workforce to match their own generation capability since they can’t depend on the grid.

  7. Finally, a good word for electric reliability!

    It started with deregulation. A classic regulated utility kept about a 20% reserve over peak load with no single unit over 10%. But when deregulation swept the politico-economic scene, margins fell to 10%. Investment in increased interconnectivity from more power transmission lines came to a near halt from NIMBY-ism and disadvantageous tax depreciation policy. Electric transmission is in the longest MACRS category.

    Add in the insanity of non-dispatchable or predictable solar and wind and reliability becomes very iffy. I’m surprised it is as good as it has been.

    Portland’s problem of a major distribution center failure is very common as maintenance and upgrade budgets are squeezed.

    But even a regulated utility and its regulator have to compromise. An example, in the 1980s in California, we wanted to upgrade the 500 kV grid with new breakers with a higher seismic capability, supplied from Japan. PG&E was denied since our state’s goal was 99.95% availablity. That’s a annual probability of 4 or 5 hours per year over many years. We could not justify the investment against that criterion given our estimate of big earthquakes although a failure of a 500 kV substation, especially on the Pacific Intertie, would have very large economic costs to the state and whole West Coast.

  8. “PG&E was denied since our state’s goal was 99.95% availablity.”

    The whole deregulation thing in California was a disaster. The Utilities were forced by legislation to sell the generating facilities.

    Typically, Wikipedia is misleading about what happened.

    Gray Davis would not allow the utilities to sign long term contracts with the generators, which the goofy legislation had forced them to surrender.

    At least this part is accurate.

    Eventually a total of 40% of installed capacity – 20 gigawatts – was sold to what were called “independent power producers.” These included Mirant, Reliant, Williams, Dynegy, and AES. The utilities were then required to buy their electricity from the newly created day-ahead only market, the California Power Exchange (PX). Utilities were precluded from entering into longer-term agreements that would have allowed them to hedge their energy purchases and mitigate day-to-day swings in prices due to transient supply disruptions and demand spikes from hot weather.

    Enron, which the article blames for the crisis was just an opportunist, like Soros was in the British Pound or short sellers are in bear markets.

    California is still in the throes of lunacy on electricity.

  9. The worst power cut we’ve had in the last couple of decades was caused by a cat shorting a transformer. The cat was called Sedgwick, and the bloody thing survived.

    We expect substantial winter power cuts soon, entirely as a consequence of stupid government policies over the last decade or so. There are limits to the extent to which the ingenuity of the engineers can overcome the folly of the pols.

  10. I worked a contract at PG&E’s headquarters back the the Nineties. I was the only heterosexual member of the tech pubs department, and I was fired for declining the advances of a male coworker. Every single electrical bill that came into the building was copied and processed and laundered in a fourteen-step process, and the final results were fed into a vintage 1950’s IBM punchcard-operated computer that, despite several million-dollar reorgs, nobody had figured out how to replace.

    A few years ago, somebody shot up a relay station in the South Bay and almost put Silicon Valley in the dark. Everybody assumed it was a terrorist or a right-wing nutjob, but I am (a) gun-literate and (b) actually worked at PG&E, so I privily came to the conclusion that it was a Les Nessman type, possibly fired or passed over because *he* would’t suck dick. He used an AK-47 loaded with ball ammo. They found one hundred spent cases, which suggest two 50-round surplus Rumanian drum magazines. He had marked the spot previously, and he knew exactly what piece of machinery to shoot, and he must have known that taking out that station would cause a cascading failure that would damage most of the power grid in the Bay Area… but he used an underpowered weapon with crappy ammo, and didn’t even bother to police his spent brass. The event was not made public for six months, by which time all sense of urgency had departed. The alarmed homeowners and companies that would have rushed out and gotten backup generators and evolved disaster-relief plans… did not.

    Ever since then, I’ve been waiting. I keep telling my godson’s prog-lib family in Redwood City to get a chest freezer, to buy a plain old single-barrel shotgun, to get a little generator with a power strip with USB terminals so they can recharge their iPhones. All they have in their earthquake kit is a small pantry in the garage with some canned good and a flat of bottled water. Apparently Uncle Phil’s doomsday BS falls on deaf ears.

    In a similar idiom, I recall after 9/11 when Spanish intelligence busted an al Qaeda cell and recovered a videotape showing targets in California. One, I recall, was the support struts of the Golden Gate Bridge. The other was the turnstiles at Disneyland. DISNEYLAND. Think about that for a moment. Think about a half a dozen young men with legal semi auto rifles, and the hundreds of families with their children wearing Mickey Mouse hats. Now we have several millions more of these barbarians living on our soil, in the name of openness and tolerance.

    I’ve been waiting for that, too. There is a lot I’m waiting for. It’s the process analyst in me.

  11. There’s an uninterruptible feedback loop that the preparedness crowd is, remarkably not using, the real estate market. Get a listing for every property and mark down each weakness in infrastructure from water supply to sewer, to electricity, to whatever else should be listed and offer an individualized property report for buyers looking to get a better deal on their house prices.

    The real estate people will adopt such a report if available. They also will pressure the system to fix it all in a politically irresistible way.

    If the infrastructure stuff is important and urgent, why aren’t the people complaining about preparedness creating that report?

    And before you ask, yes, I *am* working on such a thing but my own resources are not sufficient to fix it all.

  12. I’m uneasy about the cars, for one. I sometimes think I should get back the 1996 Nissan truck I gave my daughter. She still has it and it would probably be less susceptible to the EMP than what I have now.

    Haven’t most cars been chipped since the 1980s?

  13. Haven’t most cars been chipped since the 1980s?

    Yes, My 1986 Ford Ranger pickup had an Intel 8086 processor running things. Imagine my surprise one day as I was heading home at 50 mph on a curvy country road when it suddenly died. No engine, power steering. I managed to wrestle it to the shoulder and off the road and hiked back into town to the local mechanic I used. He towed it in and told me he already knew what the problem was. It seems that the Ford designers put the processor in the engine compartment where the heat would eventually cause it to burn out and fail. $200 for a new chip, plugged it in for another $25, and I was good to go. That processor failed twice more in the 14 years I had that truck.

  14. I’m preparing to launch the ark any day now.

    So you’re the guy I saw floating down I55 last week…….

  15. Close Joe. Not I-55, but the golden waters (not in a good way) of the mighty Des Plaines River. If we get any more rain I’ll have riverfront views from my back deck.

    Our storm sewers here are just slightly better than Bangladesh level. We were supposed to see improvements with the Deep Tunnel. It was going to be the 8th wonder of the world engineering marvel that would solve all our flooding problems, but I’m starting to think they better go deeper, much deeper. Just a few weeks ago they had to do once again what the Deep Tunnel was built to prevent, the release of raw sewage into Lake Michigan. Here we are, we’ll into the 21st century, and we’re still dealing with the same probems with water we had a century and a half ago.

  16. “what the Deep Tunnel was built to prevent,”

    Money has to be kept for graft and welfare payments to illegals.

    Do you expect to have infrastructure built with money needed by illegals and BLM ?

  17. “Haven’t most cars been chipped since the 1980s?”

    Yup, even my 87 Samurai has a chipped carb, although it’s not hard to take it completely analog. I need a Harley Carb and some bits to replace the module in my distributor.

    Now my 93 Mark VIII has OBD1 computer control and once you get past the, “oh no, it’s a computer” it works very well. I have not taken control of it but for about $300 or so I can do so. Set all the numbers, just go nuts, but it’s pretty good the way it is. They nerfed it after 93 to some extent. It’s nice to look at what happens when I hit WOT on the TPS, throttle position sensor, and it’s kinda impressive. The air conditioner gets shut down for that time, among many other things, and the whole system does it’s best to get power down.

    It’s also handy just to ask the computer what’s wrong, although I grew up figuring it out, so I usually know.

  18. From what I’ve read, EMP testing of cars/trucks has in most cases resulted in a temporary shutdown of the vehicle, but it could usually be rebooted by disconnecting and reconnecting the battery.

    There is some concern about the controllers for airbags…unlike the case with the engine controllers, a transient deployment isn’t reversible.

    Devices are particularly vulnerable to EMP when they are connected to long conductive wires, as with the electrical grid and with copper telephone wires.

  19. As it turns out my building wasn’t relying on a backup generator – they had a feed to a different substation because they were built recently and the builder took the time to plan for this to ensure a higher level of reliability. Kudos to them. It is an expensive apartment and I just renewed it and this is a perk that is highly valuable. Likely at some point in the future buildings will emphasize reliability as a selling point.

    I also believe that having a backup generator for houses, ideally one connected to a backup power source (like a natural gas line) would be another way to increase reliability, and one that should be valued by those buying a house. I heard that since you can put a lien on these that this was a great new business to get into (installing backup generators), because they can be financed and if the home owner doesn’t pay like I said above you can put a lien on the house and then when it is sold you get paid anyways.

    The concept of reliability needs to be quantified and understood by the general public. If it was, then it would be a differentiator for residents and owners alike.

    What sucks is the external costs that the utility forces on everyone through their inherent reliability or lack therof. This is a difficult problem that I don’t have any immediate ideas to fix. There is a low bar where the utility is kicked by “fines” to do better and customers complain and flee… perhaps that is enough to ensure that reliability doesn’t go too far down the toilet. But I don’t know…

  20. When I lived st the beach 30 years ago, it was during th gasoline rationing period. I put a 440 gallon diesel tank in the side yard. I did it myself with a backhoe as it was illegal.

    I had all diesel cars for a while and would fill them from the tank. I got an oil company that supplied a school bus terminal down the hill to fill y tank once a month or so.

    We had very unreliable power at the time and I gave serious thought to a diesel generator run off the tank. I didn’t do it but gave it serious thought. There were some Chinese diesel generators at the time that were about the the right size.

    My stepson in Oregon has a backup generator run off a propane tank. They are on 50 acres and have plenty of room. He has his freezer connected with a switch in case there is an outage when he is not home. The freezer is full.

Comments are closed.