Post-mortem on April 16th downtime

On April 16th I was investigating issues with the Internet Gateway and decided to run the update and let it reboot. This procedure should have resulted in less than 5 minutes of downtime. But alas, this was not the case. A corruption of the config file caused the machine to not start back up. It started to reboot at 2:15 but never finished.

After troubleshooting the issue, I decided to just wipe the machine and start over from scratch. At this time I didn’t know it was the config file that was causing the problem. I reinstalled the software and restored the configuration, resulting in the exact same problem I had before, the machine wouldn’t boot.

By this time I had figured out it was the configuration that was causing the problem, so I reinstalled yet again, but then selectively re-installed the configuration information from the backup and had the gateway back online by 4:20.

As I was heading home, I was notified that local sites weren’t working. I verified this at home, and came back in to fix it. The restore didn’t add all the IP addresses to the gateway, affecting local sites. This was fixed and everything was back to normal by 5:15.

Moral of the story, never assume a 5 minute reboot will take only 5 minutes. 🙂

Google Drive updates

Google is going to transition everyone to the new Google Drive in the next couple of months. Afterwhich, you will lose the ability to go back to the old Google Drive.

Google Apps update alerts: An update on the new Google Drive UI:

For Scheduled release customers, the rollout of the new Google Drive UI as the default user experience will occur in early April (currently planned for April 7, as per the release calendar).

In the first half of May at the earliest, we’ll remove the option to return to the old Drive UI via the Settings drop down. At least two weeks before the removal of the option to return to the old Drive UI, we will show a notification in the Drive UI about the upcoming change.

Please note that the new Drive UI does not work with older unsupported browsers prior to and including Chrome 23, Firefox 23, IE9 and Safari 6, so it’s important to upgrade to a supported browser to ensure continued access to Drive.

New Google Drive looks like this:

Screen Shot 2015-04-01 at 8.55.38 AM

Old Google Drive:

Screen Shot 2015-04-01 at 8.56.15 AM