Event Start Time: 11:40 am
Event End Time: 3:10 pm
Issue: Server failure due to Amazon S3 service outage
11:40 am The infrastructure team was alerted to an outage of SchoolObjects service. Our teams immediately began investigating the cause.
11:56 am The infrastructure team determined the outage to be a result of a nationwide Amazon outage.
12:18 pm Amazon has updated their status page to indicate they are investigating increased error rates for their servers status page: https://status.aws.amazon.com/. The S3 service, used to store files for most Amazon products, was experiencing high error rates.
1:35 pm Amazon announced they had found what they believed was the root cause of the issue and were actively working on a solution.
3:00 pm Eduphoria determined servers were beginning to come back online.
3:10 pm After a minimum of 6 servers were available, we enabled access to SchoolObjects again. Some Amazon services are still running with degraded performance.
3:43 pm Currently we suspect new file uploads could fail. This issue could affect onsite customers with Amazon backup enabled.
Our infrastructure team is discussing plans for addressing similar circumstances in the future, though our options are limited due to the robustness that Amazon provides with their cloud; nonetheless we are considering all contingency plans at this time.