Fiber Maintenance, July 11

Due to Department of Transportation-mandated construction, one of the two high-speed fibers connecting our network to the Internet will have to be relocated, and this work will take place during a window from 9 a.m. to 6 p.m. on Sunday, July 11.

This temporary outage will cause severe disruptions on our network, as the infrastructure between the towers will be likely to suffer over-congestion at several points. During this maintenance, bandwidth-intensive activities such as streaming video, videoconferencing, and gaming may be effectively unusable, and even lighter bandwidth activities, such as web browsing and email, will probably give occasional error messages.

We apologize for the inconvenience this will cause, and we hope the actual down time will be much shorter than the nine hour maintenance window.

Total Outage, June 8th

Last night (June 8th, 2021), there was scheduled maintenance at the datacenter in Dallas, where they had to de-energize each of their power distribution circuits for maintenance. The first one they de-energized, at 10:35 pm, went normally, as all of the devices we installed there have dual redundant power supplies. That circuit came back up at 11:21 pm, and no subscriber traffic had been affected by the power outage at all. The next circuit they de-energized, at 11:35 pm, had no impact on almost all of the equipment, with one exception: the edge router which connects our network to the internet.

Apparently, the second of its redundant power supplies had failed, and the specific design of this unit meant it did not report this information to us, so when it needed to rely on that power supply for power, the power supply was unable to provide any, and that router simply shut down. Unfortunately, there was nothing we could have done at that point but wait for power to be restored, which occurred at 12:27 am, and as soon as the sole working power supply in that router came back online, so too did our network reconnect to the internet.

During the outage, our phone system and email system were also unable to reach the internet, as they are located inside our network as well.

This was a bit of a freak occurrence, as the data center has power redundancy and battery backups. This specific event occurred because of routine maintenance on those redundant power systems, so it is unlikely to recur for many years, and we will be replacing that unit with one which does have working dual redundant power supplies before the next maintenance window occurs.

Oops! (mea culpa)

In normal months, accounts which are delinquent have their service deactivated in the middle of the month, typically between the 15th and the 19th. This particular February brought some record-setting weather, and has caused many Central Texas residents a great deal of trouble.

Because of all the hardships some of our subscribers may be facing, we made a decision to not deactivate service this month, but we wanted to remind those who had not made their February payment to please do so if they could, and to contact us to arrange a plan if they could not. Simple, right?

Unfortunately, the tool in our billing system we tried to leverage to send out that reminder (again, only to accounts which had an outstanding balance) would have needed a big config change to work correctly. Without that big config change, it would send that message to literally every active customer, which would be a nightmare!

As you may have now guessed, I failed to fix that configuration in advance, and thousands of messages went out, instead of the much smaller number which should have. Statistically speaking, the odds are overwhelming that you were sent the message in error. More specifically, if you did not already receive a reminder on February 8th, then you must not have had an outstanding balance at that time, and you may just delete the email. If you know you made a payment this month, just delete the email.

On the other hand, if you have not yet been able to make your payment for February service, we would ask you to contact us. We understand that some people were unable to work, and you may be experiencing a financial hardship as a result of the storm. We are willing to make arrangements so that your service may remain uninterrupted while we give you time to get caught up. If you’re in that boat, please give us a call.

The Winter Wipeout – Updates

The paralyzing winter storm, with its associated power outages, is still affecting several of our tower sites, although most are back up at this time.

As of 9:00 am on Friday, February 19, we have only one remaining tower without power, but Oncor does not have a specific ETA for repair completion.

  • Holland

At this time, if your connection is not working, please try to verify everything is interconnected in exactly the way it was before the storms, power off all your equipment, restore power to it and wait about 10 minutes. If you are still unable to connect to the internet, please call us so we can troubleshoot with you.

Because of the way our coverage area works, you may be pointed to a tower which is currently down, while your literal next-door neighbor could be pointed to a completely different tower which is currently in service. Unfortunately, we are still not able to exercise any control over this, and it’s still not completely over yet.

We have been unable to get any updates on when these sites will have power restored – neither Oncor nor PEC are providing any predicted restoration times. ERCOT is now telling local power companies they can begin to restore the sections of the grid which were shutdown because of the shortfall, but of course, there are areas which have lost power because of weather-related damage which will need to be repaired, and the extent and scale of those issues is exactly the sort of information we have no visibility to.

For our part, we have been responding to emails and dealing with equipment issues as necessary during this whole event, and we are now answering incoming phone calls again.

The Winter Wipeout

Well, the winter storm came in, as predicted, and as wind turbines and gas generators froze up all over the state, Texas power capacity was not enough to meet the demand, and ERCOT (the Texas Agency in charge of Texas’s power grid) has been trying to figure out how to handle the situation. This combined with ice build-up on power lines and all number of little local issues to put such an enormous strain on the power grid that they attempted a rolling blackout program, but the blackouts did not relieve the stress enough, and many areas were left without power for hours on end.

We have two high capacity fibers running from two nodes in our service area into a datacenter in Dallas. Unfortunately, many commercial districts of Dallas were among the areas with long power outages, and when their battery backup systems failed, our fiber interconnects were effectively severed, leaving our entire network cut off. That included our mail servers, as well as the phone system. There was literally nothing which we could do beyond reporting these fiber outages to our fiber provider, who themselves were swamped with trouble reports from the same set of circumstances, to the point where their trouble ticketing system was overwhelmed and crashed.

At this time, one of the two fibers has been restored to service. Our main problem continues to be the power grid – we have battery backup units at every tower site, but they are intended to give us a buffer until power can be restored or we can dispatch someone with a generator. They were not intended to be a long-term substitute for mains power, and many of them have become completely drained. Those are effectively offering no buffer, or, at best, only a few minutes of reserve power before the tower goes offline. The towers are, so far, working normally when they have power, so we have good confidence that full power restoration will solve nearly all of the current problems.

That’s the good news. The bad news is that some sites effectively act as hubs through which other tower’s connections are distributed, so if those towers are offline, they can impact subscribers on other towers which may not be suffering an outage. We all still seem to be dealing with frequent outages, so the grid is not yet back to normal. I’m typing this on a laptop, which is great, because I have had a good half-dozen brief outages while I typed it.

To sum up – most of the issues our subscribers are currently facing are entirely power-based, and we believe that once the power grid is brought back to normal functionality, the vast majority of our subscribers should come back online without any intervention. If you do not currently have service, we do apologize for this inconvenience, and we assure you we are working to get everyone back up as soon as possible.

UPDATE: 10:52 am, Tuesday morning: As of about 10:15 am, the second of our fibers has been restored, so the entirety of problems at present is due to power outages at the various tower sites.

UPDATE: noon on Wednesday, February 17: Our tower sites still remain at the mercy of the power companies, but both fibers have been up without incident for over 24 hours, and we have been able to get some brief contact with most of our sites while the power was briefly up. This was very encouraging – we are not finding a whole lot of equipment which is having any issue other than these power outages. I’ve detected some weird glitches with some components, but I have been able to remotely reset things as needed and get nearly all of them back working normally.

As per KXAN, we are expecting more wintry precipitation tonight and tomorrow, but the current forecasts suggest that the worst of it may stay south of most of our service area. Please stay as safe and as warm as you can, boil water before consuming it if your water provider advises it (and your water is running), and be extremely careful about using non-electric heaters indoors, as they can generate lethal carbon monoxide gas.

Fiber Outage

A little after 6:30 this morning (February 11, 2021), one of our two fiber feeds to the internet went offline. The fiber provider has identified the issue and is working on repairs, but we don’t yet have an ETA on the repairs.

What does this mean for you, our customers? Right now, all our internet traffic is attempting to redirect over backup paths to the functioning fiber, but these paths do not have the capacity to handle the full level of the excess traffic. You will be seeing spotty performance for any tasks until it’s resolved.

This also impacts Voice over IP phone service, including our phone system, resulting in choppy audio and dropped calls. That is part of the reason why the phones are not configured to send calls to us at this time – there’s no point, as the audio quality is so bad, and we are not able to speed the repair process along in any way.

I will update this post when I have more information.

UPDATE, 10:16 am: The fiber circuit has been repaired and traffic flow is back to normal.

Merry Christmas, and Happy Gaming!

As I type this, Christmas is just two weeks away, and one of the most anticipated games in years, Cyberpunk 2077, has just been released earlier this week.

This seems like a good time to remind our subscribers that this season of giving, specifically giving video games, may result in some concerns with your family’s internet performance.

People who purchase this game for PC, for instance, have to download the massive game files (over 60GB, plus a Day One patch which is also many GB in size), but even console players can’t just drop the disc in their machine and play, as the required Day One patch for Cyberpunk 2077 on consoles is typically 25-30 GB.

To put this into perspective, this table shows the best-case scenario for download times of a 25 GB file (or 214,748,364,800 bits) at each of our plans’ speeds. By “best case”, I mean under perfect circumstances:

  1. a flawless connection between the device and the router (because every data packet which must be re-sent adds extra time)
  2. literally no other traffic through the router whatsoever, so the download can utilize 100% of the capacity of your internet connection
  3. servers on the publisher’s side which have the capacity to serve all packets at maximum speed, without any bottlenecks anywhere along the path from their servers to our datacenter, which could inadvertently limit the speed

(Any variance from perfection equals a longer download time.)

Download SpeedNumber of hours needed
318.96
69.48
96.32
124.74
153.79

As members of your household open their presents on Christmas morning and pop their new video games into their systems, pretty much all of these games will require updates in order to play. Some needed updates may be pretty small, but others could be very large, indeed.

Please be patient during this process, and do understand that it may be necessary, if your household wishes to utilize the internet normally on Christmas Day, to wait until everyone is done with the internet for the day or ready to go to bed before attempting to install the new games, and that, depending on the specific details, these games may not be ready to play for quite a while. (If you have the ability and time, you could theoretically set up the console and install the games and patches in the remaining time before Christmas Day, sticking to the middle of the night for downloads, which would allow for play as soon as the console is unwrapped.)

We also want to remind everyone that many gaming consoles, including Nintendo Switch, Xbox One and Series X, and PlayStation 4 and 5 consoles, can (and do) use the internet connection to download data, even when the consoles are not in use and/or appear to be off. If the console’s primary user cannot tell the console to pause the downloads, it may be necessary to disconnect the power completely from the console to get it to stop monopolizing your internet connection.

Also, please note the units used in these calculations: Our service’s speed plans (as is the case with most Internet Service Providers) are expressed in megabits-per-second values, but it’s not unusual for systems to display in-progress file transfer speeds in terms of bytes (or kilobytes, megabytes, or gigabytes) per second, and as each byte is made up of 8 bits, this can be a little confusing. If the speed display you see uses a lowercase “b”, it probably means bits, and if it uses an uppercase “B”, it’s probably bytes. All figures in terms of bytes/time should be multiplied by 8 to convert the value to bits/time.

Sales Tax Law Changes

At the end of last week, we were informed that there had been a recent change in the law regarding sales tax collection for internet service, taking effect as of July 1st. Under the old laws, the first $25 of internet service fees were exempt from sales tax, but the remainder of the invoices were all subject to sales tax. Under the new law change, all internet access fees are now exempt from sales tax.

Based on our conversations with the Texas Comptroller’s Office, our understanding is that this exemption removes all sales tax from regular internet service fees, and any other services which directly affect internet access, which in this case refers to fees for a fixed public IP address. Router rentals involve a physical object, and setup fees are primarily labor and are separate from internet service fees, so the taxation rules on those did not change. Similarly, web hosting and domain registration fees are covered in a separate section of the sales tax laws, and were not affected by this.

Of course, as I mentioned, we were not told about this in advance, so we had been calculating and collecting tax for these invoices normally. In order to make this transition simpler and fairly pain-free for our subscribers, our CEO decided to just credit everyone’s accounts for the sales tax levied against all applicable invoices, so none of you will need to file paperwork with the state tax office to get these taxes refunded – we are taking care of that.

These credits have already been applied to all affected accounts, and you do not need to do anything. You can log into the payment portal and view each of the credit invoices which has been generated for your account. Each credit invoice has a description line which refers to the original invoice on which the tax was charged.

If you have any questions about this, please do not call us on the telephone about this. Please email us about any questions or concerns about these tax issues at billing@ecpi.com. As this is essentially a billing issue, we don’t want to overwhelm our billing team with questions, some of which may not be answerable by them in the first place. Also, please remember the billing team is available Monday-Friday, 8 am to 6 pm, and use of the after-hours tech support line for non-technical support purposes, such as this, will result in a $10 fee being applied to your account

Moving forward, we have already changed the settings in the system, so all future invoices should reflect the updated tax laws.

Fiber cut

As most of our subscribers have probably noted, the fiber cut we suffered at about 10:30 this morning is having a big impact on nearly all our subscribers at this time. We did file a trouble ticket with the fiber provider as soon as the event occurred, and they are working on tracking down the damage and repairing it, but we do not yet have a ETA for repair completion.

I will be updating this blog entry as more information comes in.


As of 3:17 pm, the splicing team is on-site and assessing the damage.


The current estimated time of repair is 8:00 pm, CDT


The fiber repair has been completed, and traffic is now flowing normally. The storm which is currently passing through the service area is creating an impact, too, but that should be short-lived.

We apologize for this inconvenience, and thank you for your patience.

Western Broadband Update, 3-23-2020

The global COVID-19 pandemic is having a big impact on everyone’s lives. We encourage all of our subscribers to be aware of the risks, not just to ourselves, but also to the more vulnerable members of our communities here in Central Texas, and to take all necessary precautions to slow the spread of this disease.

Bandwidth Usage

In order to “flatten the curve” and prevent our hospitals and healthcare systems from being overwhelmed, most schools have sent students home, and many businesses are encouraging or even requiring their employees to work from home. This combination has already resulted in higher-than-normal bandwidth demands throughout the day, and as more people shift to this new, temporary way-of-life, we expect to see that increase even further.

Our normal planning method for increasing capacity looks at existing demand and extrapolates based on a simple, linear growth model. This new surge in demand was unexpected, and although we are constantly upgrading and improving our network, and we did already have plans to add additional capacity to support higher speed plans and improve redundancy, the new equipment for this is here at our office, still awaiting FCC license approvals, at which point we will need to schedule tower climbs to install it.

We fully intend to try to expedite this process as much as possible, but all this additional bandwidth demand is stressing our network in new ways. As a result, there may be periods of the day during which you may not be receiving all the network speed you normally expect. Just like the grocery stores had their shelves cleared of hand sanitizer and bleach and have to scramble to restock, we are working hard to improve our infrastructure, but it can’t be done all at once.

We ask for some patience, and request that you please refrain from running speedtests, if possible – speedtests do not have some secret process to objectively determine the speed of a connection; they work by throwing as much data as they can at our system, and detecting the point at which the connection chokes. That means that when you think your connection is running slow, and you run a speedtest, you are putting even more traffic, carried over the whole path from the fiber to your access point and then your home, possibly during those same periods in which everyone else on the same access point is competing to use bandwidth. If you are concerned that you’re not getting the speed you should be, please call us on the telephone, and we will be happy to run the necessary diagnostics to figure out what is happening. That process may involve a speedtest, but under monitored conditions.

If you require additional bandwidth to meet your work-from-home needs, or to allow your children to use learning-from-home software, please call us. We can evaluate the quality of your connection, and determine if we can make the requested increase. In some cases, a site visit may be required to improve the signal level in order to allow the higher-performance plans to work properly. In certain instances, where the signal levels are already at the best we can manage, yet are still marginal, we may not be able to increase the speed as much as desired.

Special WiFi Hotspot

We know that the wireless connections we offer are limited in speed by their nature. We are working on improving speeds to your homes, as I mentioned above, but we already have a high-speed fiber optic line which connects to our office in Austin. During this crisis, we understand that you may need to move files so large that they won’t work well over your home’s connection, so we are setting up a WiFi Hotspot accessible from the parking lot at our office. You can use a laptop and see upload and download speeds much faster than you’d get at home for those few files that are too big to send or receive from home. Please call or email us for details.