[FQuest Alert] MySQL14 Server

[Ryan]

MySQL14 has gone down and is being powercycled at this time.

We will post again as soon as we have additional information.

Our apologies,
Ryan

[Ryan]

The MySQL14 server has been brought back up at this time.

Event Start: 11:16 AM ET
Event End: 11:30 AM ET
Duration 14 minutes

[Kevin]

We think the MYSQL14 server suffered a disk failure and then crashed. The failure may have been temporary or it might not we will know when the RAID rsync either finishes or fails. We are also doing a MySQL check/repair on any tables that have problems.

[Kevin]

The checks and repairs have completed. The RAID resync is at 30%.

[Kevin]

The RAID resync has finished. That means there is no dead disk. Hopefully the crash was just a fluke.

[Kevin]

Something is still not right. The RAID suddenly decided to start resyncing again without an error.

[Kevin]

The RAID has failed 3 times now. It was not the same disk so I believe the disks are actually fine. Something must be causing the RAID controller to flake out. Could be the controller itself or even the power supply. I will have to go take the server down and investigate. I am running a quick backup too.

[Mike]

The MySQL14 went down again briefly and was powercycled before returning to service.

Event Start: 8:09 PM ET
Event End: 8:14 PM ET
Duration 5 minutes

Kevin is investigating the issue.

[Kevin]

It appears the server has failed due to the dreaded bad capacitors problem. I have the hardware working again but I have hit a corruption problem that I am trying to figure out.

[Kevin]

The server is back up. This was the first time I ever encountered a broken lvm2 pv personally but luckily I found out years ago to make sure the lvm2 backup files (in /etc/lvm) are not dependent on lvm2 working correctly.

The database repair process is back underway.