MYSQL15 server problems

**Kevin** · 09-02-2021, 10:37 AM

We are currently experiencing intermittent problems with the MYSQL15 server. It is showing multiple symptoms so we are having difficulty determining the root cause.

**Kevin** · 09-02-2021, 12:04 PM

The check/repair process has completed and the MySQL15 server is currently working properly. We still don't have a diagnosis but we have some ideas. We have turned on some additional monitoring to hopefully catch the problem if it happens again.

We also have emailed a site owner that was doing something unusual during the last crash. We don't have much info about what they were doing but have essentially asked for details of what they were doing and why and more importantly were they doing it during the other crashes. This is pretty important as the last crash was the worst and had additional symptoms that the previous crashes didn't have which of course makes diagnostics far more complicated.

On a side note... When there is a MySQL crash whether it is MySQL itself or the whole server our standard procedure it to run 'mysqlcheck -A --auto-repair' This is needed because any table that was open at the time needs to at least be checked and marked clean and at worst be repaired. Every time I do this I see stuff scroll by with names that are obviously obsolete stuff that has just been kept around and forgotten about. Even if a database hasn't been touched in years it still has to be checked. This takes time. The mysqlcheck program checks in alphabetic order then repairs the problems it found in the same order. This means that anything actually broken stays broken until after all of the checking happens. I certainly understand the desire to keep old databases around after a major upgrade (especially long enough for our nightly backups to back them up) but please don't just forget about them.

**Kevin** · 09-06-2021, 11:26 AM

These problems went away for the weekend but they are back this morning. We are again trying to determine the cause of the problems.

**Kevin** · 09-06-2021, 03:41 PM

I have been working on this all day and I haven't been able to determine much. The behavior is completely inconsistent. I have come to the conclusion that something must be wrong with the physical server. So, I am on my way in to investigate.

**Kevin** · 09-06-2021, 04:33 PM

Well, this is a new one for me...

Apparently the power supply partially failed. The fan in it was not spinning and it was hot. I swapped it out with a spare and the server is up and running again. I plugged the PSU into our tester and the lights just blinked.

**Kevin** · 09-06-2021, 05:02 PM

It seems I spoke too soon. Something is still wrong.

**Kevin** · 09-06-2021, 05:40 PM

OK, I have now replaced the rest of the server. The replacement hardware is up and it is doing the check/repair process again. The replacement hardware is an upgrade.

**Kevin** · 09-06-2021, 06:17 PM

The check/repair process has finally finished. I do need to take it back offline for a few minutes to fix a filesystem issue. This won't be a crash though so it should come up normally when done.

**Kevin** · 09-06-2021, 06:32 PM

OK, the MYSQL15 server is now back in full production. All known errors have been fixed. We apologize for the extended nature of this outage. Unfortunately I have no idea what was wrong with the other system. It is possible that the failing power supply damaged the electronics. I will see if I can do some diagnostics on it later.