Western Broadband

Anyone using an @ecpi.com or @pmwifi.com email address has probably noticed the issues with the mail server which got pretty bad toward the end of April. The most common issue was getting timeout errors when trying to log in to check or send messages, which would prompt Outlook to ask for the password again.

UPDATE: May 9 –
This is the set of instructions which seem to work best for updating Outlook:

• With Outlook running and in the foreground, press Ctrl-Alt-S
  You should see the “Send/Receive Groups” dialog, with the “All Accounts” line highlighted
• Hit Edit, then select the account and hit the “Account Properties” button.
  You should now be on the “Internet E-mail Settings” window.
• On that first tab, make sure that both mail servers are set to “mail.ecpi.com”, and that the username shows as username@ecpi.com
• Click on the “Advanced” tab, and make sure that the “This server requires an encrypted connection (SSL)” under the incoming server is checked on.
  The incoming server number should now be 995 for POP or 993 for IMAP
• Also make sure that the “Use the following type of encrypted connection” under the outgoing server is set to either “TLS” or “SSL/TLS” – whichever of these options is available.
•  Then click OK, OK, and close. Now, when you send and receive, it should work normally.
  
  

There were some log files which had swelled over the years, and there were some other huge configuration files which were inducing delays during most login attempts. We have fixed those problems, and updated the mail server and its Operating System version, and it appears to be working better than it has in years. That’s the good news…

The bad news is that some settings have been changed, and this seems to be causing an issue with Microsoft Outlook for some subscribers. It seems that phones and tablets are typically not having the issues, just Outlook.

In order to fix Outlook, you might need to get into the Account Settings in Outlook. Older versions, up through Outlook 2010, have this accessible from the File Menu under Account Settings, while newer versions of Outlook hide that information under the Control Panel. Here is a link to a Microsoft Support page, explaining how to access this on the newer versions:

Once there, you’ll need to turn encryption on for both Incoming Mail and Outgoing Mail – it should look something like this for POP*:

You may also need to add the domain name to the email address if you had previously only used the first part as your User Name, like so:

*: For IMAP, the advanced settings should look like this:

5/4: UPDATE: Some people have been running into a issue where Outlook complains that it cannot save the settings. Here’s what Tom figured out:

We had problems trying to make the changes to Outlook to use SSL on the incoming POP mail server.

Outlook said “ something went wrong – could not save new settings”

We found a fix:

If you’re trying to make your changes from File -> Account settings, try doing it from Send & Receive settings instead.

1. From inside Outlook, Press Ctrl+Alt+S
2. Click Edit
3. Select/click the account you want to update, then click “Account Properties”

Make your changes from there and you should avoid the “Something went wrong…” message and be able to save your changes and access your email again. 

This worked!

Doing the changes from File > Account settings did NOT work.
Doing the changes from Control Panel did NOT work.

The above process worked immediately. We did not have to reboot, or even restart Outlook.

We will be updating our phone server during the middle of the night (1 am to 8 am ) on July 6th, 2022. During this time, we do expect that the phone server may be offline for periods of time, and no one will be able to reach our after-hours support by telephone. We do not expect any further issues beyond that.

We discovered, around 11:30 today, that our phone trunk provider is experiencing problems. These problems make it nearly impossible for the outside world to call into any of our telephone numbers. It had also affected our outgoing calling ability, but I have figured out a way to work around that. Subscribers who have our phone system can call out now, but incoming calls, by and large, seem to be unable to be completed at this time.

The nature of the problem seems to be that the authoritative nameserver used by this trunk provider is inaccessible, but the nature of the problem also means we cannot easily reach out to the company to communicate these problems or to get feedback on when the problem will be fixed.

I will update this space once I have more information or the problem has been resolved.

As of 1 pm, incoming calls (at least from the Verizon network) are working, although calls from T-Mobile are not.

3:00 pm – The trunk provider has posted an update, and seems to have been able to work around the issue. It seems to have been caused by a deliberate denial of service attack against them. They report that they “have made changes in the configuration with [their] upstream carriers to mitigate the issue for incoming calls as well.”, so incoming calls should be working normally at this time for all carriers.

Friday morning, 9/17 , the problems started up again. Now, it’s even worse than it was yesterday. The trunk provider’s entire infrastructure appears to be deliberately targeted, and they are working to solve the problems and restore phone service, but they do not have an estimated time of repairs as yet.

Saturday, September 18: This problem has still not been resolved – ii you can get through on the phones, please do so, but if the phones are not working, please email our billing team at billing@ecpi.com or our support team at support@ecpi.com with any concerns.

Wednesday. September 22: Our trunk provider, VoIP.ms, is still suffering from the ongoing DDoS attacks, and we have decided to switch trunk providers to a company which does have a stable connection. We expect this transition to complete on Friday, and all phone services should be working normally by then.

Thursday, September 23: The transition to the new trunk provider has occurred ahead of schedule, and our phone system should now be working normally. Please let me know (at support@ecpi.com) if you are a VoIP customer and you are still having any issues, or if you need any changes to your Caller ID.

Due to Department of Transportation-mandated construction, one of the two high-speed fibers connecting our network to the Internet will have to be relocated, and this work will take place during a window from 9 a.m. to 6 p.m. on Sunday, July 11.

This temporary outage will cause severe disruptions on our network, as the infrastructure between the towers will be likely to suffer over-congestion at several points. During this maintenance, bandwidth-intensive activities such as streaming video, videoconferencing, and gaming may be effectively unusable, and even lighter bandwidth activities, such as web browsing and email, will probably give occasional error messages.

We apologize for the inconvenience this will cause, and we hope the actual down time will be much shorter than the nine hour maintenance window.

Last night (June 8th, 2021), there was scheduled maintenance at the datacenter in Dallas, where they had to de-energize each of their power distribution circuits for maintenance. The first one they de-energized, at 10:35 pm, went normally, as all of the devices we installed there have dual redundant power supplies. That circuit came back up at 11:21 pm, and no subscriber traffic had been affected by the power outage at all. The next circuit they de-energized, at 11:35 pm, had no impact on almost all of the equipment, with one exception: the edge router which connects our network to the internet.

Apparently, the second of its redundant power supplies had failed, and the specific design of this unit meant it did not report this information to us, so when it needed to rely on that power supply for power, the power supply was unable to provide any, and that router simply shut down. Unfortunately, there was nothing we could have done at that point but wait for power to be restored, which occurred at 12:27 am, and as soon as the sole working power supply in that router came back online, so too did our network reconnect to the internet.

During the outage, our phone system and email system were also unable to reach the internet, as they are located inside our network as well.

This was a bit of a freak occurrence, as the data center has power redundancy and battery backups. This specific event occurred because of routine maintenance on those redundant power systems, so it is unlikely to recur for many years, and we will be replacing that unit with one which does have working dual redundant power supplies before the next maintenance window occurs.

In normal months, accounts which are delinquent have their service deactivated in the middle of the month, typically between the 15th and the 19th. This particular February brought some record-setting weather, and has caused many Central Texas residents a great deal of trouble.

Because of all the hardships some of our subscribers may be facing, we made a decision to not deactivate service this month, but we wanted to remind those who had not made their February payment to please do so if they could, and to contact us to arrange a plan if they could not. Simple, right?

Unfortunately, the tool in our billing system we tried to leverage to send out that reminder (again, only to accounts which had an outstanding balance) would have needed a big config change to work correctly. Without that big config change, it would send that message to literally every active customer, which would be a nightmare!

As you may have now guessed, I failed to fix that configuration in advance, and thousands of messages went out, instead of the much smaller number which should have. Statistically speaking, the odds are overwhelming that you were sent the message in error. More specifically, if you did not already receive a reminder on February 8th, then you must not have had an outstanding balance at that time, and you may just delete the email. If you know you made a payment this month, just delete the email.

On the other hand, if you have not yet been able to make your payment for February service, we would ask you to contact us. We understand that some people were unable to work, and you may be experiencing a financial hardship as a result of the storm. We are willing to make arrangements so that your service may remain uninterrupted while we give you time to get caught up. If you’re in that boat, please give us a call.

The paralyzing winter storm, with its associated power outages, is still affecting several of our tower sites, although most are back up at this time.

As of 9:00 am on Friday, February 19, we have only one remaining tower without power, but Oncor does not have a specific ETA for repair completion.

• Holland

At this time, if your connection is not working, please try to verify everything is interconnected in exactly the way it was before the storms, power off all your equipment, restore power to it and wait about 10 minutes. If you are still unable to connect to the internet, please call us so we can troubleshoot with you.

Because of the way our coverage area works, you may be pointed to a tower which is currently down, while your literal next-door neighbor could be pointed to a completely different tower which is currently in service. Unfortunately, we are still not able to exercise any control over this, and it’s still not completely over yet.

We have been unable to get any updates on when these sites will have power restored – neither Oncor nor PEC are providing any predicted restoration times. ERCOT is now telling local power companies they can begin to restore the sections of the grid which were shutdown because of the shortfall, but of course, there are areas which have lost power because of weather-related damage which will need to be repaired, and the extent and scale of those issues is exactly the sort of information we have no visibility to.

For our part, we have been responding to emails and dealing with equipment issues as necessary during this whole event, and we are now answering incoming phone calls again.

Well, the winter storm came in, as predicted, and as wind turbines and gas generators froze up all over the state, Texas power capacity was not enough to meet the demand, and ERCOT (the Texas Agency in charge of Texas’s power grid) has been trying to figure out how to handle the situation. This combined with ice build-up on power lines and all number of little local issues to put such an enormous strain on the power grid that they attempted a rolling blackout program, but the blackouts did not relieve the stress enough, and many areas were left without power for hours on end.

We have two high capacity fibers running from two nodes in our service area into a datacenter in Dallas. Unfortunately, many commercial districts of Dallas were among the areas with long power outages, and when their battery backup systems failed, our fiber interconnects were effectively severed, leaving our entire network cut off. That included our mail servers, as well as the phone system. There was literally nothing which we could do beyond reporting these fiber outages to our fiber provider, who themselves were swamped with trouble reports from the same set of circumstances, to the point where their trouble ticketing system was overwhelmed and crashed.

At this time, one of the two fibers has been restored to service. Our main problem continues to be the power grid – we have battery backup units at every tower site, but they are intended to give us a buffer until power can be restored or we can dispatch someone with a generator. They were not intended to be a long-term substitute for mains power, and many of them have become completely drained. Those are effectively offering no buffer, or, at best, only a few minutes of reserve power before the tower goes offline. The towers are, so far, working normally when they have power, so we have good confidence that full power restoration will solve nearly all of the current problems.

That’s the good news. The bad news is that some sites effectively act as hubs through which other tower’s connections are distributed, so if those towers are offline, they can impact subscribers on other towers which may not be suffering an outage. We all still seem to be dealing with frequent outages, so the grid is not yet back to normal. I’m typing this on a laptop, which is great, because I have had a good half-dozen brief outages while I typed it.

To sum up – most of the issues our subscribers are currently facing are entirely power-based, and we believe that once the power grid is brought back to normal functionality, the vast majority of our subscribers should come back online without any intervention. If you do not currently have service, we do apologize for this inconvenience, and we assure you we are working to get everyone back up as soon as possible.

UPDATE: 10:52 am, Tuesday morning: As of about 10:15 am, the second of our fibers has been restored, so the entirety of problems at present is due to power outages at the various tower sites.

UPDATE: noon on Wednesday, February 17: Our tower sites still remain at the mercy of the power companies, but both fibers have been up without incident for over 24 hours, and we have been able to get some brief contact with most of our sites while the power was briefly up. This was very encouraging – we are not finding a whole lot of equipment which is having any issue other than these power outages. I’ve detected some weird glitches with some components, but I have been able to remotely reset things as needed and get nearly all of them back working normally.

As per KXAN, we are expecting more wintry precipitation tonight and tomorrow, but the current forecasts suggest that the worst of it may stay south of most of our service area. Please stay as safe and as warm as you can, boil water before consuming it if your water provider advises it (and your water is running), and be extremely careful about using non-electric heaters indoors, as they can generate lethal carbon monoxide gas.

A little after 6:30 this morning (February 11, 2021), one of our two fiber feeds to the internet went offline. The fiber provider has identified the issue and is working on repairs, but we don’t yet have an ETA on the repairs.

What does this mean for you, our customers? Right now, all our internet traffic is attempting to redirect over backup paths to the functioning fiber, but these paths do not have the capacity to handle the full level of the excess traffic. You will be seeing spotty performance for any tasks until it’s resolved.

This also impacts Voice over IP phone service, including our phone system, resulting in choppy audio and dropped calls. That is part of the reason why the phones are not configured to send calls to us at this time – there’s no point, as the audio quality is so bad, and we are not able to speed the repair process along in any way.

I will update this post when I have more information.

UPDATE, 10:16 am: The fiber circuit has been repaired and traffic flow is back to normal.

As I type this, Christmas is just two weeks away, and one of the most anticipated games in years, Cyberpunk 2077, has just been released earlier this week.

This seems like a good time to remind our subscribers that this season of giving, specifically giving video games, may result in some concerns with your family’s internet performance.

People who purchase this game for PC, for instance, have to download the massive game files (over 60GB, plus a Day One patch which is also many GB in size), but even console players can’t just drop the disc in their machine and play, as the required Day One patch for Cyberpunk 2077 on consoles is typically 25-30 GB.

To put this into perspective, this table shows the best-case scenario for download times of a 25 GB file (or 214,748,364,800 bits) at each of our plans’ speeds. By “best case”, I mean under perfect circumstances:

1. a flawless connection between the device and the router (because every data packet which must be re-sent adds extra time)
2. literally no other traffic through the router whatsoever, so the download can utilize 100% of the capacity of your internet connection
3. servers on the publisher’s side which have the capacity to serve all packets at maximum speed, without any bottlenecks anywhere along the path from their servers to our datacenter, which could inadvertently limit the speed

(Any variance from perfection equals a longer download time.)

Download Speed	Number of hours needed
3	18.96
6	9.48
9	6.32
12	4.74
15	3.79

As members of your household open their presents on Christmas morning and pop their new video games into their systems, pretty much all of these games will require updates in order to play. Some needed updates may be pretty small, but others could be very large, indeed.

Please be patient during this process, and do understand that it may be necessary, if your household wishes to utilize the internet normally on Christmas Day, to wait until everyone is done with the internet for the day or ready to go to bed before attempting to install the new games, and that, depending on the specific details, these games may not be ready to play for quite a while. (If you have the ability and time, you could theoretically set up the console and install the games and patches in the remaining time before Christmas Day, sticking to the middle of the night for downloads, which would allow for play as soon as the console is unwrapped.)

We also want to remind everyone that many gaming consoles, including Nintendo Switch, Xbox One and Series X, and PlayStation 4 and 5 consoles, can (and do) use the internet connection to download data, even when the consoles are not in use and/or appear to be off. If the console’s primary user cannot tell the console to pause the downloads, it may be necessary to disconnect the power completely from the console to get it to stop monopolizing your internet connection.

Also, please note the units used in these calculations: Our service’s speed plans (as is the case with most Internet Service Providers) are expressed in megabits-per-second values, but it’s not unusual for systems to display in-progress file transfer speeds in terms of bytes (or kilobytes, megabytes, or gigabytes) per second, and as each byte is made up of 8 bits, this can be a little confusing. If the speed display you see uses a lowercase “b”, it probably means bits, and if it uses an uppercase “B”, it’s probably bytes. All figures in terms of bytes/time should be multiplied by 8 to convert the value to bits/time.