Service Interrruption, Multiple Servers Affected

**Mike** · 08-29-2021, 12:55 PM

We appear to have a switch or circuit down, Techs are currently investigating the issue. This is affecting the TAZ, DEXTER, HUGO, XIS, EVA, SIX, ASTRO, DEEDEE, ZAT, REI and WALLE Community servers. All MQS dedicated servers are unaffected.

We will post as soon as services have been brought back up, and will post further details and updates as they come in.

Our apologies,
Mike

**Mike** · 08-29-2021, 01:10 PM

Attempts to restore services remotely have met with no success. Preliminary investigation suggests we lost a power source, or (hopefully) just the breaker to it. Kevin is en route to the data center now.

We will continue to report any updates as the information comes in.

**Mike** · 08-29-2021, 01:34 PM

Kevin arrived shortly ago and services to the affected community servers are coming back up now. The issue is that the breaker on a power controller had failed.

Event Start: 12:42 PM EST
Event End: 1:31 PM EST

Duration: 49 minutes

**Kevin** · 08-29-2021, 03:25 PM

If anyone is curious this was the current version of the power controller:
https://www.apc.com/shop/us/en/produ...ckets/P-AP8932
It has 2 breakers that control the top and bottom halves. The one for the top half had tripped. We have 3 of those particular models of power controller and this happened to us once before but on a different one.

Essentially, we have no reason to believe this was anything but a fluke on an otherwise reliable piece of infrastructure.