On April 16th I was investigating issues with the Internet Gateway and decided to run the update and let it reboot. This procedure should have resulted in less than 5 minutes of downtime. But alas, this was not the case. A corruption of the config file caused the machine to not start back up. It started to reboot at 2:15 but never finished.
After troubleshooting the issue, I decided to just wipe the machine and start over from scratch. At this time I didn’t know it was the config file that was causing the problem. I reinstalled the software and restored the configuration, resulting in the exact same problem I had before, the machine wouldn’t boot.
By this time I had figured out it was the configuration that was causing the problem, so I reinstalled yet again, but then selectively re-installed the configuration information from the backup and had the gateway back online by 4:20.
As I was heading home, I was notified that local sites weren’t working. I verified this at home, and came back in to fix it. The restore didn’t add all the IP addresses to the gateway, affecting local sites. This was fixed and everything was back to normal by 5:15.
Moral of the story, never assume a 5 minute reboot will take only 5 minutes. 🙂