Network Outage

[Mike]

This morning at 9:54 ET the FutureQuest network went down following a major power outage at the data center. The outage was a local one, and utility company personnel were on site investigating the issue when one of our techs arrived minutes later.

Approximately two hours ago the power was restored, at which time Techs began rebooting the core network routers.

An hour ago several services reestablished and came back online, and since then techs have been bringing up the community and dedicated servers, MYSQL servers and email. Forums were restored shortly ago, but techs are still troubleshooting and rebooting certain servers. We hope to have all services restored very soon.

Once everything has been restored to normal, a technician will post here with a full report.

Our apologies for the inconvenience, and thank you for your patience.

[Wrightwood]

4-24-2022 outage
Couple of questions that come to mind...
Didn't FQ replace a bunch of batteries after the last outage?
Isn't the backup generator cycled weekly?

[Grant]

We were down for over 6 hours, possibly as many as 10, site is now up but still no mail.

[ricktrip]

Not having info on what is happening is a bit disconcerting, but your blogspot and Facebook pages have not been updated in years. Is there another source when the Forum is down too?

[ricktrip]

Still no mail for me either.

[Grant]

I was also VERY concerned when there was NO updates to the Facebook or Twitter pages, despite assurances that there would be last time there was an outage. I even tried calling, and just got a fast busy signal, which was actually better as it meant there was a general outage of some kind.
So yes, details will be appreciated, and an improvement to communication is CLEARLY NEEDED.

[Mike]

Batteries and the backup generator will be discussed in the post-op.

We posted an original notice and updates to our twitter feeds and facebook page this morning.

Services like email and a few of the MySQL servers are still in the process of being restored, we will post here when all services are back to normal.

[wilkoff]

No email!

[Terra]

Bootstrapping an entire data center from a full cold outage, however this time around the new cold bootstrap orchestration has helped quite a bit in smoothing out the proper startup ordering.

We currently have most all services restored except for the following:
MX Load Balancers (causing email to be offline)
MYSQL07
MYSQL08
MYSQL10

These servers have had their power supplies blown out from surge and we are working on a solution. The main issue is the tech supply chain has been slow as molasses and/or out of stock/backorder and the spare power supplies are not in yet.

The MX Load Balancers (MXLB), are what is causing email to be offline as they front and balance all backend email traffic to the POP Toasters. We are currently working on a virtualized workaround so that I can easily shift this functionality in case of a primary failure. Due to it being such a critical piece of backend infrastructure, we've been hesitant to virtualize this because the underlying CNI networking required component was still in beta. I have taken another look and it appears the required component is now ready for prime time. We are currently making the necessary changes to the core switches, and running preliminary tests to see if it will work reliably. Once that is deemed reliable, the new images will be built and wired in to the network.

Our sincerest apologies for the outage and we are working as fast (hair on fire) as humanly possible to resolve the remaining issues. Thankfully full cold outages are incredibly rare, but if it should ever rear its ugly head, it usually releases a handful of unforeseen/unpredictable gremlins that don't want to be caught and we have to capture them on a case by case basis.

[MarkW]

I had to look at 3 twitter feeds before I found posts at:
https://twitter.com/FutureQuest

The other 2 feeds:
https://twitter.com/FutureQuestNet
https://twitter.com/FQStatus