Nobody knew how to restore power at Ullrich Water Treatment Plant during the freeze. It was out for three hours
On a normal day, Ullrich Water Treatment Plant produces roughly half of Austin's drinkable water and is crucial to keeping the city's water system functioning.
State regulations require the plant to either have access to a backup power source or a substantial amount of water reserves in case the plant sees an unexpected shutdown. Ullrich has both.
So when a tree limb fell on an electric line leading to a substation that powered Austin's largest water treatment plant on Feb. 17, backups should have snapped into place to keep power running and water production churning.
But there was a problem: Nobody on site knew how to operate a 52-year-old gear switch that would have restored power to the plant.
And so Ullrich Water Treatment Plant went dark for three hours in the middle of the worst winter storm to strike Central Texas in decades. It cut off roughly half of the city's potable water production and deepened the winter weather crisis that at that moment had thousands shivering without electricity in their homes.
The outage resulted in the plant operating below capacity for more than 10 hours, according to Austin Water. During that time, water drained rapidly from the city's water reserves, bled by thousands of burst pipes and dozens of broken water mains.
The failure of dual redundancies at Ullrich helped trigger a citywide boil water notice and contributed to widespread loss of water pressure that included forcing the evacuation of many patients from St. David's South Austin Medical Center.
In the weeklong winter storm, the loss of power at Ullrich represents an inflection point when the city's power crisis became a water supply crisis.
Ullrich had a second emergency power source and enough water storage to provide 100 gallons to each customer in case of a power outage. Yet the icy conditions revealed that Ullrich Water Treatment Plant's dual redundancies — backup power and backup water storage — were unreliable when faced with weather conditions during the winter storm.
It was "literally a perfect storm," Austin Water's assistant director of operations, Rick Coronado, said in an interview with the American-Statesman.
Broken water mains and pipes rapidly began depleting the Austin's reserve water stored in massive reservoirs even before the outage. So when the power went out, there was little water left in reserve to keep the system pressurized enough to push water out of faucets, bathtubs and spigots.
"The outage compounded an already difficult situation of keeping up with leaks and system demand that ultimately depressurized the water system," Coronado said.
It is still unclear the level of impact the power outage at Ullrich Water Treatment Plant had on the overall water system. The city has not yet conducted an after-action report, and hearings have only just begun on what actually happened.
But its impact was undeniable. Ullrich accounts for about half of Austin Water's capacity for treating the water that the utility's three treatment plants pull in from Lake Austin and Lake Travis.
'Water use was exploding'
Even before the power went out at the Ullrich plant, big problems were already facing Austin’s water system.
The city issued a boil water advisory for Lost Creek and some neighborhoods in Southwest Austin at 11 a.m. Feb. 17. At nearly that exact moment, water usage peaked at roughly 250 million gallons, more than double the usual amount Austinites use daily.
"Water use was exploding," Austin Water Director Greg Meszaros told the City Council this past week. "It was just climbing off the dial for us. Quite honestly, I didn't believe the numbers, and I ordered some of our crews to go and physically check the reservoirs, particularly in the southwest where this was first starting, and verify that these numbers were correct. As we all learned, they were correct."
At roughly 2:30 p.m. – a point when about 35% of the city was still without power – the on-site power substation at Ullrich went offline. The water treatment plant has a backup power connection to Austin Energy.
Tom Pierpoint, Austin Energy's vice president of electric systems, said the power utility monitored the outage to the substation from its control center. Operators were able to restore power to the substation within six minutes by isolating the damaged portion of the substation, Pierpoint said.
Yet power was not restored to Ullrich Water Treatment Plant for roughly three hours, according to Austin Water. Coronado said no one at the site could operate a manual gear switch installed in 1969 that could have switched the plant to backup power.
The exact sequence of events seemed to be unclear to both Austin Energy and Austin Water officials. Coronado said Austin Water staffers were not trained to switch the water treatment plant to a backup source of electricity – while Pierpoint said Austin Water should have known how to conduct such a task.
Regardless, the plant remained offline for about three hours while Austin Water officials waited for an Austin Energy crew to arrive to flip the switch.
"For them to get out to the water plant with treacherous roads in such bad times, it was a challenge to get out there," Pierpoint said.
Once power was restored at Ullrich, the plant had to be eased into full operations, a process that took about six hours, according to Austin Water.
During the time Ullrich was not at full production capacity, St. David's South Austin Medical Center saw its water pressure drop to a point that medical officials ordered some patient evacuations because the hospital's heating system relied on water boilers that no longer had adequate pressure. Austin Water issued a citywide boil water notice that night, at about 9 p.m.
About six hours later – in the early morning of Feb. 18 – Austin's water reserves had fallen to 12.5 million gallons, roughly one-tenth of the amount the utility aims to keep in reserves, according to Austin Water.
Over the next five days, Austin Water worked to restore water storage and bring water pressure to the whole system. That began with pressurizing portions of the water system for Central and East Austin, because they are closely connected to water treatment plants and are in the lowest elevation portion of the city's service area.
As those pipes pressurized, Austin Water crews worked to fix hundreds of water main breaks. During this past Thursday's City Council meeting, Meszaros said the water utility repaired roughly 40 to 60 breaks a day. The total number of water main breaks in the single weather event was roughly equivalent to the number Austin Water would see for an entire season.
Water service was restored to the full system this past Monday. On Tuesday, the boil water notice was lifted. And on Thursday, Austin Water lifted water use restrictions. Despite that, many Austin residents remain without water. On Thursday, Meszaros estimated that 200 to 400 apartment and condo complexes continue to lack running water because of burst pipes.
'We must do better'
Many questions remain about what exactly went wrong at Ullrich Water Treatment Plant and the extent its shutdown contributed to the water crisis. Some answers might come during a joint meeting of the City Council's two utility oversight committees that is scheduled for Wednesday.
"The council and the public need answers," Council Member Alison Alter said Friday in a posting to the council's online message board. "We need to understand where our systems fell short and how they were so vulnerable."
There are already indications the city was aware that the Ullrich Water Treatment Plant's backup system needed an upgrade before the storm. The 52-year-old gear switches at the plant are set to be replaced with automated switches as part of $20 million of upgrades, which the utility financed in January, according to city documents. A presentation on the upgrades to Ullrich's electrical system indicated they were needed to "ensure the plant can reliably deliver drinking water."
Decades-old electrical equipment at Davis Water Treatment Plant, Austin Water's second-largest treatment plant, also is slated for replacement, Coronado said. Handcox Water Treatment Plant, the system's newest plant, has automated switches.
Nhat M. Ho, a civil engineer and member of the city's Water and Wastewater Commission, which has oversight over Austin Water, said it is hard to lay blame on Austin Water given the extraordinary extremes of the winter storm.
"Sometimes people would say, 'Well why don't you design for it,'" Ho said. "That's like saying, 'Why don't you make every car bulletproof.'"
Ho said that Austin Water might need to set new benchmarks for planning given the likely increase in frequency of severe weather events expected with climate change.
"A lot of people are hurting, so a lot of this feels irrelevant right now, but at the same time, I think it's fair for the folks to keep that in mind," he said.
But William Moriarty, another member of the Water and Wastewater Commission, said he needs to hear more about what happened at Ullrich Water Treatment Plant.
"If it is true that the city failed to correctly operate the switching mechanism to engage the alternative power source, then that is gross incompetence," Moriarty said. "We must do better."