[FQuest Notice] SAN work progress updates - Page 2

**Terra** · 12-09-2020, 08:53 AM

esmtim, In 5 minutes, everything should be back to normal - I'm doing a reset of the Pop Toasters just to be safe and clear out any held locks.

*adixon* · 12-09-2020, 09:21 AM

Can't login to Questmail. 9:20AM EST

Can login now but messages in Inbox not displaying 9:23 AM

**Terra** · 12-09-2020, 09:26 AM

IMAP services are currently having a lock contention problem and we are working as quickly as possible to prepare for a full reset.

**Terra** · 12-09-2020, 09:56 AM

We are still waiting for full convergence before we can do a full reset of the mail system. The cause for the delay is the old storage kernel driver OOPs'd earlier and I installed an updated version that should fix the hard lock problem. The downside is when it came back up, one of its tasks is to update all the storage chunks to current version. This is what is consuming the time as this silo holds about 2M chunks and I have to let it run its course to ensure there is no data loss whatsoever.

It was bad timing for the driver to OOPs, and it was one of the last servers I had to upgrade but didn't have enough time this morning to get it done before I ran into the end of my overnight maintenance window. I was hoping it wouldn't hard lock before I had a chance to upgrade it tomorrow night - but it did. Luckily I had all the upgrades ready and in place - just needed to pull the trigger. Sadly, it came 18 hours before it should have.

As soon as this task clears, I will do a full mail system reset to ensure all services have a fresh start. Also - all emails are accounted for and there was no data loss.

*daniel77* · 12-09-2020, 09:59 AM

So, should this be the end of all of the email issues, or is there more to come from the upgrades and security fixes?

*Marbledog* · 12-09-2020, 10:14 AM

Wow thank you Terra - I went from "omg no email again" and imagining 2 days of chaotic hell, to reading your posts and feeling relieved knowing that it's being handled. Communication makes life a lot less stressful on our end.

**Terra** · 12-09-2020, 10:28 AM

Ok - the storage chunk updates to current version is taking way longer than I'd like. I am going to attempt something rather tricky to pull off to get the mail system reset. In theory, what I have planned should work, but since I've never had to do a leapfrog like this - I just don't know. FWIW, it won't cause any worse issues than what there is now. (crossing fingers)

During this procedure, I'm shutting down all email systems (communications) while I perform this (never been tried) advanced technique. No email will be lost, it will retry and send once PT servers are back online.

**Terra** · 12-09-2020, 10:29 AM

Quote:

Originally Posted by daniel77

So, should this be the end of all of the email issues, or is there more to come from the upgrades and security fixes?

It was the last storage system that needed an upgraded kernel driver. I tried to get it done earlier this morning, but ran out of time due to maintenance window closing.

**Terra** · 12-09-2020, 10:49 AM

Phase 1 of the procedure is done which was ensuring server comes up in minimal mode - no services running.

2 or 3 more to go before full reset is achieved while in a degraded mode.

PT reboots next up

**Kevin** · 12-09-2020, 10:54 AM

A note since several people have asked...

If you are getting certificate errors those are unrelated to this problem. The certificates were coincidentally updated last night. If you are getting these errors it means your email client is configured incorrectly.

If you want to use encrypted email (which of course we recommend) and you get these errors change your incoming and outgoing email server names to mail.questmail.net instead of the domain specific names. Also, use the full email address as the user name.

If you manually force your email client to accept the new certificate instead of making the change you will be right back to the same problem in a month with the certificates renew again.