“S3 Is Down” is never something any IT manager expected to hear but, on February 28, that is exactly what happened, turning the day into an interesting–and unplanned–test of our core infrastructure.
Over the past several years, we have moved more and more of our critical GTP and SKUpics infrastructure to AWS which has enabled us to rapidly scale and offer high-availability services. Fortunately we still prioritize on-site and redundant assets meaning that our companies can weather a storm, even one as big as “the cloud” shutting down.
While some customer services were affected for a few hours, GTP’s on-site customer-facing systems across the country remained functional throughout the outage and handled the Amazon failure in stride, properly caching and recovering following the AWS restoration.
At SKUpics, we were able to lean on our software’s proprietary “offline mode,” a capability designed to allow us to be fully operational in difficult shooting environments where network and power–or “the cloud”–are not always present.
Between our support team monitoring and responding to the event in Hershey, PA and our fault-tolerant on-site infrastructure, “The Day S3 Went Down” will be little more than a blip on our radar. Because of information we learned from how our systems responded to the outage, we’ll be able to mitigate future disruptions even further, making for an even stronger GTP and SKUpics.
The cloud may have shut down, but it’s always sunny at GTP and SKUpics!
Be sure to check out the full postmortem of the S3 event from Amazon.