Oops! (mea culpa)

In normal months, accounts which are delinquent have their service deactivated in the middle of the month, typically between the 15th and the 19th. This particular February brought some record-setting weather, and has caused many Central Texas residents a great deal of trouble.

Because of all the hardships some of our subscribers may be facing, we made a decision to not deactivate service this month, but we wanted to remind those who had not made their February payment to please do so if they could, and to contact us to arrange a plan if they could not. Simple, right?

Unfortunately, the tool in our billing system we tried to leverage to send out that reminder (again, only to accounts which had an outstanding balance) would have needed a big config change to work correctly. Without that big config change, it would send that message to literally every active customer, which would be a nightmare!

As you may have now guessed, I failed to fix that configuration in advance, and thousands of messages went out, instead of the much smaller number which should have. Statistically speaking, the odds are overwhelming that you were sent the message in error. More specifically, if you did not already receive a reminder on February 8th, then you must not have had an outstanding balance at that time, and you may just delete the email. If you know you made a payment this month, just delete the email.

On the other hand, if you have not yet been able to make your payment for February service, we would ask you to contact us. We understand that some people were unable to work, and you may be experiencing a financial hardship as a result of the storm. We are willing to make arrangements so that your service may remain uninterrupted while we give you time to get caught up. If you’re in that boat, please give us a call.

The Winter Wipeout – Updates

The paralyzing winter storm, with its associated power outages, is still affecting several of our tower sites, although most are back up at this time.

As of 9:00 am on Friday, February 19, we have only one remaining tower without power, but Oncor does not have a specific ETA for repair completion.

  • Holland

At this time, if your connection is not working, please try to verify everything is interconnected in exactly the way it was before the storms, power off all your equipment, restore power to it and wait about 10 minutes. If you are still unable to connect to the internet, please call us so we can troubleshoot with you.

Because of the way our coverage area works, you may be pointed to a tower which is currently down, while your literal next-door neighbor could be pointed to a completely different tower which is currently in service. Unfortunately, we are still not able to exercise any control over this, and it’s still not completely over yet.

We have been unable to get any updates on when these sites will have power restored – neither Oncor nor PEC are providing any predicted restoration times. ERCOT is now telling local power companies they can begin to restore the sections of the grid which were shutdown because of the shortfall, but of course, there are areas which have lost power because of weather-related damage which will need to be repaired, and the extent and scale of those issues is exactly the sort of information we have no visibility to.

For our part, we have been responding to emails and dealing with equipment issues as necessary during this whole event, and we are now answering incoming phone calls again.

The Winter Wipeout

Well, the winter storm came in, as predicted, and as wind turbines and gas generators froze up all over the state, Texas power capacity was not enough to meet the demand, and ERCOT (the Texas Agency in charge of Texas’s power grid) has been trying to figure out how to handle the situation. This combined with ice build-up on power lines and all number of little local issues to put such an enormous strain on the power grid that they attempted a rolling blackout program, but the blackouts did not relieve the stress enough, and many areas were left without power for hours on end.

We have two high capacity fibers running from two nodes in our service area into a datacenter in Dallas. Unfortunately, many commercial districts of Dallas were among the areas with long power outages, and when their battery backup systems failed, our fiber interconnects were effectively severed, leaving our entire network cut off. That included our mail servers, as well as the phone system. There was literally nothing which we could do beyond reporting these fiber outages to our fiber provider, who themselves were swamped with trouble reports from the same set of circumstances, to the point where their trouble ticketing system was overwhelmed and crashed.

At this time, one of the two fibers has been restored to service. Our main problem continues to be the power grid – we have battery backup units at every tower site, but they are intended to give us a buffer until power can be restored or we can dispatch someone with a generator. They were not intended to be a long-term substitute for mains power, and many of them have become completely drained. Those are effectively offering no buffer, or, at best, only a few minutes of reserve power before the tower goes offline. The towers are, so far, working normally when they have power, so we have good confidence that full power restoration will solve nearly all of the current problems.

That’s the good news. The bad news is that some sites effectively act as hubs through which other tower’s connections are distributed, so if those towers are offline, they can impact subscribers on other towers which may not be suffering an outage. We all still seem to be dealing with frequent outages, so the grid is not yet back to normal. I’m typing this on a laptop, which is great, because I have had a good half-dozen brief outages while I typed it.

To sum up – most of the issues our subscribers are currently facing are entirely power-based, and we believe that once the power grid is brought back to normal functionality, the vast majority of our subscribers should come back online without any intervention. If you do not currently have service, we do apologize for this inconvenience, and we assure you we are working to get everyone back up as soon as possible.

UPDATE: 10:52 am, Tuesday morning: As of about 10:15 am, the second of our fibers has been restored, so the entirety of problems at present is due to power outages at the various tower sites.

UPDATE: noon on Wednesday, February 17: Our tower sites still remain at the mercy of the power companies, but both fibers have been up without incident for over 24 hours, and we have been able to get some brief contact with most of our sites while the power was briefly up. This was very encouraging – we are not finding a whole lot of equipment which is having any issue other than these power outages. I’ve detected some weird glitches with some components, but I have been able to remotely reset things as needed and get nearly all of them back working normally.

As per KXAN, we are expecting more wintry precipitation tonight and tomorrow, but the current forecasts suggest that the worst of it may stay south of most of our service area. Please stay as safe and as warm as you can, boil water before consuming it if your water provider advises it (and your water is running), and be extremely careful about using non-electric heaters indoors, as they can generate lethal carbon monoxide gas.

Fiber Outage

A little after 6:30 this morning (February 11, 2021), one of our two fiber feeds to the internet went offline. The fiber provider has identified the issue and is working on repairs, but we don’t yet have an ETA on the repairs.

What does this mean for you, our customers? Right now, all our internet traffic is attempting to redirect over backup paths to the functioning fiber, but these paths do not have the capacity to handle the full level of the excess traffic. You will be seeing spotty performance for any tasks until it’s resolved.

This also impacts Voice over IP phone service, including our phone system, resulting in choppy audio and dropped calls. That is part of the reason why the phones are not configured to send calls to us at this time – there’s no point, as the audio quality is so bad, and we are not able to speed the repair process along in any way.

I will update this post when I have more information.

UPDATE, 10:16 am: The fiber circuit has been repaired and traffic flow is back to normal.

Merry Christmas, and Happy Gaming!

As I type this, Christmas is just two weeks away, and one of the most anticipated games in years, Cyberpunk 2077, has just been released earlier this week.

This seems like a good time to remind our subscribers that this season of giving, specifically giving video games, may result in some concerns with your family’s internet performance.

People who purchase this game for PC, for instance, have to download the massive game files (over 60GB, plus a Day One patch which is also many GB in size), but even console players can’t just drop the disc in their machine and play, as the required Day One patch for Cyberpunk 2077 on consoles is typically 25-30 GB.

To put this into perspective, this table shows the best-case scenario for download times of a 25 GB file (or 214,748,364,800 bits) at each of our plans’ speeds. By “best case”, I mean under perfect circumstances:

  1. a flawless connection between the device and the router (because every data packet which must be re-sent adds extra time)
  2. literally no other traffic through the router whatsoever, so the download can utilize 100% of the capacity of your internet connection
  3. servers on the publisher’s side which have the capacity to serve all packets at maximum speed, without any bottlenecks anywhere along the path from their servers to our datacenter, which could inadvertently limit the speed

(Any variance from perfection equals a longer download time.)

Download SpeedNumber of hours needed
318.96
69.48
96.32
124.74
153.79

As members of your household open their presents on Christmas morning and pop their new video games into their systems, pretty much all of these games will require updates in order to play. Some needed updates may be pretty small, but others could be very large, indeed.

Please be patient during this process, and do understand that it may be necessary, if your household wishes to utilize the internet normally on Christmas Day, to wait until everyone is done with the internet for the day or ready to go to bed before attempting to install the new games, and that, depending on the specific details, these games may not be ready to play for quite a while. (If you have the ability and time, you could theoretically set up the console and install the games and patches in the remaining time before Christmas Day, sticking to the middle of the night for downloads, which would allow for play as soon as the console is unwrapped.)

We also want to remind everyone that many gaming consoles, including Nintendo Switch, Xbox One and Series X, and PlayStation 4 and 5 consoles, can (and do) use the internet connection to download data, even when the consoles are not in use and/or appear to be off. If the console’s primary user cannot tell the console to pause the downloads, it may be necessary to disconnect the power completely from the console to get it to stop monopolizing your internet connection.

Also, please note the units used in these calculations: Our service’s speed plans (as is the case with most Internet Service Providers) are expressed in megabits-per-second values, but it’s not unusual for systems to display in-progress file transfer speeds in terms of bytes (or kilobytes, megabytes, or gigabytes) per second, and as each byte is made up of 8 bits, this can be a little confusing. If the speed display you see uses a lowercase “b”, it probably means bits, and if it uses an uppercase “B”, it’s probably bytes. All figures in terms of bytes/time should be multiplied by 8 to convert the value to bits/time.

Sales Tax Law Changes

At the end of last week, we were informed that there had been a recent change in the law regarding sales tax collection for internet service, taking effect as of July 1st. Under the old laws, the first $25 of internet service fees were exempt from sales tax, but the remainder of the invoices were all subject to sales tax. Under the new law change, all internet access fees are now exempt from sales tax.

Based on our conversations with the Texas Comptroller’s Office, our understanding is that this exemption removes all sales tax from regular internet service fees, and any other services which directly affect internet access, which in this case refers to fees for a fixed public IP address. Router rentals involve a physical object, and setup fees are primarily labor and are separate from internet service fees, so the taxation rules on those did not change. Similarly, web hosting and domain registration fees are covered in a separate section of the sales tax laws, and were not affected by this.

Of course, as I mentioned, we were not told about this in advance, so we had been calculating and collecting tax for these invoices normally. In order to make this transition simpler and fairly pain-free for our subscribers, our CEO decided to just credit everyone’s accounts for the sales tax levied against all applicable invoices, so none of you will need to file paperwork with the state tax office to get these taxes refunded – we are taking care of that.

These credits have already been applied to all affected accounts, and you do not need to do anything. You can log into the payment portal and view each of the credit invoices which has been generated for your account. Each credit invoice has a description line which refers to the original invoice on which the tax was charged.

If you have any questions about this, please do not call us on the telephone about this. Please email us about any questions or concerns about these tax issues at billing@ecpi.com. As this is essentially a billing issue, we don’t want to overwhelm our billing team with questions, some of which may not be answerable by them in the first place. Also, please remember the billing team is available Monday-Friday, 8 am to 6 pm, and use of the after-hours tech support line for non-technical support purposes, such as this, will result in a $10 fee being applied to your account

Moving forward, we have already changed the settings in the system, so all future invoices should reflect the updated tax laws.

Fiber cut

As most of our subscribers have probably noted, the fiber cut we suffered at about 10:30 this morning is having a big impact on nearly all our subscribers at this time. We did file a trouble ticket with the fiber provider as soon as the event occurred, and they are working on tracking down the damage and repairing it, but we do not yet have a ETA for repair completion.

I will be updating this blog entry as more information comes in.


As of 3:17 pm, the splicing team is on-site and assessing the damage.


The current estimated time of repair is 8:00 pm, CDT


The fiber repair has been completed, and traffic is now flowing normally. The storm which is currently passing through the service area is creating an impact, too, but that should be short-lived.

We apologize for this inconvenience, and thank you for your patience.

Western Broadband Update, 3-23-2020

The global COVID-19 pandemic is having a big impact on everyone’s lives. We encourage all of our subscribers to be aware of the risks, not just to ourselves, but also to the more vulnerable members of our communities here in Central Texas, and to take all necessary precautions to slow the spread of this disease.

Bandwidth Usage

In order to “flatten the curve” and prevent our hospitals and healthcare systems from being overwhelmed, most schools have sent students home, and many businesses are encouraging or even requiring their employees to work from home. This combination has already resulted in higher-than-normal bandwidth demands throughout the day, and as more people shift to this new, temporary way-of-life, we expect to see that increase even further.

Our normal planning method for increasing capacity looks at existing demand and extrapolates based on a simple, linear growth model. This new surge in demand was unexpected, and although we are constantly upgrading and improving our network, and we did already have plans to add additional capacity to support higher speed plans and improve redundancy, the new equipment for this is here at our office, still awaiting FCC license approvals, at which point we will need to schedule tower climbs to install it.

We fully intend to try to expedite this process as much as possible, but all this additional bandwidth demand is stressing our network in new ways. As a result, there may be periods of the day during which you may not be receiving all the network speed you normally expect. Just like the grocery stores had their shelves cleared of hand sanitizer and bleach and have to scramble to restock, we are working hard to improve our infrastructure, but it can’t be done all at once.

We ask for some patience, and request that you please refrain from running speedtests, if possible – speedtests do not have some secret process to objectively determine the speed of a connection; they work by throwing as much data as they can at our system, and detecting the point at which the connection chokes. That means that when you think your connection is running slow, and you run a speedtest, you are putting even more traffic, carried over the whole path from the fiber to your access point and then your home, possibly during those same periods in which everyone else on the same access point is competing to use bandwidth. If you are concerned that you’re not getting the speed you should be, please call us on the telephone, and we will be happy to run the necessary diagnostics to figure out what is happening. That process may involve a speedtest, but under monitored conditions.

If you require additional bandwidth to meet your work-from-home needs, or to allow your children to use learning-from-home software, please call us. We can evaluate the quality of your connection, and determine if we can make the requested increase. In some cases, a site visit may be required to improve the signal level in order to allow the higher-performance plans to work properly. In certain instances, where the signal levels are already at the best we can manage, yet are still marginal, we may not be able to increase the speed as much as desired.

Special WiFi Hotspot

We know that the wireless connections we offer are limited in speed by their nature. We are working on improving speeds to your homes, as I mentioned above, but we already have a high-speed fiber optic line which connects to our office in Austin. During this crisis, we understand that you may need to move files so large that they won’t work well over your home’s connection, so we are setting up a WiFi Hotspot accessible from the parking lot at our office. You can use a laptop and see upload and download speeds much faster than you’d get at home for those few files that are too big to send or receive from home. Please call or email us for details.

Changes!

“All is flux, nothing stays still”
― Plato


For quite some time, Western Broadband has offered in-house evening and weekend technical support. The way that worked was we had an experienced team member (Craig) who was able to answer phones 7 days a week, during every weekday evening and on weekends during the day, with relatively few breaks for holidays, vacations, or family events. He is now in a position where he is able to step down from this job and spend more time with his family. We’ll miss him, but we understand this is a good thing for him.

Of course, this leads, inevitably, to the question of how we will be handling technical support on the weekends and evenings. Hiring a new person or people to take over from Craig was considered, but it is simply not feasible. The costs of hiring and training, not to mention the time period during which any new employee would take to get a handle on everything, makes that option either very expensive (and those costs would have to be passed on to our subscribers), very inefficient, or both.

We decided our best option was to contract out our extended hours technical support needs to a new, US-based, remote office team. They have staff who already handle this sort of thing for other Internet Service Providers across the nation, and they are well experienced in solving the same sort of problems we already solve on a day-to-day basis. We believe this is the best way to ensure our subscribers’ needs are met without having to raise our rates.


How will this affect you?

  • For now, our support line hours will remain unchanged. The new service will operate M-F, 6pm to 10pm, 9 am to 5 pm on Saturdays, and noon to 5 pm on Sundays, just like Craig did. We are still not certain on how each holiday will affect the support schedule, but we can notify our subscribers of those relevant details here, in this blog.
  • Our network monitoring will not be affected. The same staff who currently monitor all the major aspects of our network, and are automatically notified when there are any problems, will continue to do this. Repairs to equipment in our datacenter or at the towers will still be handled the way they always have been, and with the same emphasis on minimizing our subscribers’ down-time.
  • We will have daily, in-house, Tier 2 escalation management. In other words, if you call in on Friday night with a problem that is outside of the scope of the evening team, you will be contacted the next day by one of our in-house personnel for additional troubleshooting, and at that time, if we need to schedule a service call, we can do that.
  • IMPORTANT: This new line is intended to be used only for technical support calls. Their staff will not be able to provide in-depth information to potential customers, so we will not be passing sales calls onto them. By the same token, they cannot answer questions about billing, or resolve billing disputes, and they will not be able to take payments or handle other, non-technical, customer support issues. Our payment portal will still be operational 24 x 7 to make payments, and telephone payments can still be made during normal operating hours, 8 am to 6 pm, Monday through Friday. Questions about your bill can be asked during normal operating hours, or emailed to billing@ecpi.com at any time.
  • IMPORTANT: Because this new support system charges per call, and we need to keep those costs down for everybody, we must charge a fee of $10 to subscribers who use this new line to call about things it is not meant to or able to handle: things like customer service issues, scheduling issues, billing questions or making payments – as mentioned above, they cannot help with billing questions, they can’t take payments, they can’t resolve customer service disputes, and they aren’t able to rearrange scheduled appointments, but we would still be charged for those calls. We have an outgoing message which will remind callers of this policy, and this system requires a button push before it transfers over to them, so we don’t expect to have too many people accidentally incurring this fee.
  • If you are uncertain about which department you need to speak with, you can always email your question to support@ecpi.com, or call us during normal operating hours.
  • As a general rule, you should NOT call the technical support line after normal hours for these topics:
    1. To make a payment or discuss anything to do with payments.
    2. To make or update a payment arrangement.
    3. To ask a question or discuss a dispute about a bill you’ve received from us.
    4. To change the credit card information we have on file for you.
    5. To change the bandwidth plan you are subscribed to.
    6. To schedule an appointment for equipment to be installed or moved at your site.
    7. To ask about an upcoming appointment which has already been scheduled.
    8. To re-schedule or cancel an appointment.
    9. To cancel service.
    10. To address a (non technical) customer service-related issue.
  • You should call the technical support line after normal hours if you:
    1. Are unable to connect to the Internet.
    2. Are connecting to the Internet, but not getting the level of performance you should be (after you’ve eliminated other devices using bandwidth).
    3. Are experiencing intermittent connection issues.
    4. Have a specific, technical problem with our connection I did not list.

Of course, this is a transition for us as much as it is a transition for you, and we do appreciate your patience as we work through this. We would also appreciate feedback from you – how clear our messages are, how effective the technical support team is, and anything else you might think is important – of course, that feedback should either be sent via email to support@ecpi.com or you can call us during those normal operating hours, 8 am to 6 pm, Monday through Friday.

To reboot or not to reboot?

To reboot or not to reboot, that is the question… (apologies to the Bard!)

Because the radio transceiver we installed at your home and your wireless router are both computers, there are situations in which one or both might lock up, crash, or refuse to communicate with the other. For that reason, it is often the case that the first thing we suggest when troubleshooting is to power-cycle one or both of these devices. When we are dealing with a “one-off” sort of problem, that usually suffices to clear it up, and your connection is back up and working normally in a few minutes.

However, if you have been experiencing some sort of repeated problem, whether it happens at the same time each day, or is intermittent or unpredictable, those troubleshooting steps are not recommended, even if they seem to grant some relief. Here’s why:  We do monitor your transceiver for certain important telemetry data (like signal level and errors), but the system simply takes instantaneous snapshots every so many minutes – useful to show a trend, but not detailed enough to do any deep inspection. Other statistical data, invaluable in determining the root cause of problems, are stored in the transceiver in volatile memory which is cleared whenever it is rebooted, whether by a power-cycle or a software command. Similarly, the historical data we retain on your overall bandwidth usage is captured every few minutes and averaged, so it can sometimes be helpful, but it’s not precise enough to help with troubleshooting many brief or intermittent problems.

The best way for us to get a full understanding of the nature of those brief or intermittent problems is to view the data in real-time when the problem is happening, and the second-best way is for us to be able to view the transceiver’s statistical data immediately after the incident. What this means in the real world is that if you have been experiencing some sort of recurring Internet problem, whether it’s intermittent, transient, or regular, you will want to give us a call on the telephone, preferably while the failure mode is ongoing, so that we can collect enough data to ascertain root cause, and you should do this without power-cycling anything. That is the best strategy to eliminate those sorts of problems.

On the other hand, if you almost never have any problems with your connection, and it’s acting up with no obvious reason, then resetting the power to the transceiver and/or router may be the quickest way to get you back online. Just remember, if you are having problems with your Internet connection, we want to help you!