The Winter Wipeout

Well, the winter storm came in, as predicted, and as wind turbines and gas generators froze up all over the state, Texas power capacity was not enough to meet the demand, and ERCOT (the Texas Agency in charge of Texas’s power grid) has been trying to figure out how to handle the situation. This combined with ice build-up on power lines and all number of little local issues to put such an enormous strain on the power grid that they attempted a rolling blackout program, but the blackouts did not relieve the stress enough, and many areas were left without power for hours on end.

We have two high capacity fibers running from two nodes in our service area into a datacenter in Dallas. Unfortunately, many commercial districts of Dallas were among the areas with long power outages, and when their battery backup systems failed, our fiber interconnects were effectively severed, leaving our entire network cut off. That included our mail servers, as well as the phone system. There was literally nothing which we could do beyond reporting these fiber outages to our fiber provider, who themselves were swamped with trouble reports from the same set of circumstances, to the point where their trouble ticketing system was overwhelmed and crashed.

At this time, one of the two fibers has been restored to service. Our main problem continues to be the power grid – we have battery backup units at every tower site, but they are intended to give us a buffer until power can be restored or we can dispatch someone with a generator. They were not intended to be a long-term substitute for mains power, and many of them have become completely drained. Those are effectively offering no buffer, or, at best, only a few minutes of reserve power before the tower goes offline. The towers are, so far, working normally when they have power, so we have good confidence that full power restoration will solve nearly all of the current problems.

That’s the good news. The bad news is that some sites effectively act as hubs through which other tower’s connections are distributed, so if those towers are offline, they can impact subscribers on other towers which may not be suffering an outage. We all still seem to be dealing with frequent outages, so the grid is not yet back to normal. I’m typing this on a laptop, which is great, because I have had a good half-dozen brief outages while I typed it.

To sum up – most of the issues our subscribers are currently facing are entirely power-based, and we believe that once the power grid is brought back to normal functionality, the vast majority of our subscribers should come back online without any intervention. If you do not currently have service, we do apologize for this inconvenience, and we assure you we are working to get everyone back up as soon as possible.

UPDATE: 10:52 am, Tuesday morning: As of about 10:15 am, the second of our fibers has been restored, so the entirety of problems at present is due to power outages at the various tower sites.

UPDATE: noon on Wednesday, February 17: Our tower sites still remain at the mercy of the power companies, but both fibers have been up without incident for over 24 hours, and we have been able to get some brief contact with most of our sites while the power was briefly up. This was very encouraging – we are not finding a whole lot of equipment which is having any issue other than these power outages. I’ve detected some weird glitches with some components, but I have been able to remotely reset things as needed and get nearly all of them back working normally.

As per KXAN, we are expecting more wintry precipitation tonight and tomorrow, but the current forecasts suggest that the worst of it may stay south of most of our service area. Please stay as safe and as warm as you can, boil water before consuming it if your water provider advises it (and your water is running), and be extremely careful about using non-electric heaters indoors, as they can generate lethal carbon monoxide gas.