Three Points on One Interconnection

The Trace

It was just an average Wednesday morning in the Dominion power system. Then at 7:56:10 AM EDT, a transmission line tripped in the Dominion power system. That sentence should end right there. There is nothing terribly unusual about a line trip. But it initiated events that caused a disturbance that rolled through the entire Eastern Interconnection. It did not trip anything, but it showed up on meters as far away as Chicago.

Robert Orndorff, a part-time energy consultant, pulled this trace off of his home frequency monitoring system. We owe Robert a thank you for sharing it with us, because this trace is the heart of this story.

Here’s the trace:

Why It Went Everywhere

This chart shows system frequency immediately increasing to almost 60.1 Hz directly after the event. This is not the first time this has happened because of a line fault in the Northern Virginia area. What is causing this is almost 3,000 MW of data center load switching off because the UPS systems that protect the computer systems detected a voltage dip, a “brownout” due to the line fault. To protect the computers, those UPS systems immediately switch to battery and disconnect from utility. The issue is that there was no real power outage; the voltage dip lasted only a few cycles, but it was enough to trip the protection. The power system simply cannot absorb 3,000 MW of lost load in a single step without causing problems. The problem it caused was high frequency, and frequency is virtually the same across the entire Eastern Interconnection.

Confirmed in Ohio

This was verified by historical frequency data I pulled from my own frequency monitor I have under development here at my home. As you can see from this trace, the frequency not only went high in Virginia, but it went high in Ohio as well (UTC Time).

Enough to Make Reuters

The disturbance showed up on enough instruments to make Reuters. A piece came out after 6 PM detailing all the places the disturbance showed up. Orndorff caught it in Virginia. I caught it, later, in Ohio. Turns out we weren’t the only ones. Ting Labs runs a network of 1.4 million home sensors, and their CEO, Bob Marshall, told Reuters they picked up voltage disturbances reaching from the Eastern Seaboard all the way to Chicago. A grid disturbance normally corrects itself in milliseconds. This one took about ten minutes to fully settle, and Ting’s own customers in northern Virginia reported flickering lights and noisy air conditioners while it did. That’s three separate instruments, at three very different scales, all telling the same story. The message is: if a simple line trip can cause this much disturbance, one needs to ask what happens if there is a bigger fault in that Northern Virginia area.

NERC Isn’t Waiting

NERC isn’t waiting to find out. On May 4^th, they issued a Level 3 NERC Alert, brought about largely from the last disturbance in this same region. NERC is looking for ways to mitigate this issue, and they want feedback by August 3^rd. This is just the first step in their action plan. Mitigation is never easy and rarely a straight path.

An Object Lesson

In many ways this mirrors my own frequency station development. The reason I pulled data from history and didn’t post my own report is that my frequency station missed this event. It triggered, but I killed the report by restarting the machine to fix another problem. Once we (Claude and I) pulled the historical data and ran it through the configuration we had, a whole basket of problems came to light, and we completely rebuilt the report engine. That’s how development goes, and how it will go for NERC.

I wanted to seize this event as an object lesson while we had the chance. Operating a power system is not easy. New problems come up all the time. We learn, we fix, and we move on. Tell us about your development story.