Issue: Intermittent degraded response time across all applications.
Status: Under Investigation. Issue temporarily resolved with restore of previous online testing and test proctoring user interface.
Issue: Errors when accessing Aware and Strive applications attributed to erroring front-end servers.
Event Start Time: 2019-08-16, 10:41 am CST
Event End Time: 2019-08-16, 3:14 pm CST
At 10:41 am - First email notification of performance issues from districts sent to firstname.lastname@example.org regarding Aware and other applications.
At 11:26 am - Eduphoria Development team released fix to resolve issues with Aware Online Testing performance.
At 12:26 pm - Continued reports of performance issues from districts across multiple applications.
At 12:36 pm - Eduphoria Infrastructure alerted Support and Engineering teams to issues with database connections.
At 1:34 pm - Eduphoria Infrastructure and Engineering teams identified potential issue causing performance degradation. The team confirmed a database connection leak was responsible for the majority of the connection issues. The connection leak had already been identified on August 8, and a fix is already in testing.
At 1:44 pm - First report of error within Strive when accessing application.
At 1:50 pm - Eduphoria decided to temporarily disable the new online testing and test proctoring UI while the fix is tested. This will allow districts to continue testing and evaluating students while the API is improved.
At 1:52 pm - First report of error within Aware when accessing district assessments.
At 2:18 pm - Previous user interface for online testing and test proctoring restored within Aware.
At 2:19 pm - Eduphoria Development team investigating cause of errors within Strive and Aware.
At 3:14 pm - Eduphoria Infrastructure team identified source of errors throughout Strive and Aware. Intermittent errors attributed to 2 erroring front-end servers. Servers were replaced within the pool.
At 4:00 pm - Continued performance monitoring by Eduphoria Support and Infrastructure teams.