Event Start Time: 8:05 am
Event End Time: 11:13am
Issue: Intermittent degraded response time across Strive, Aware, Forethought.
At 8:05 am - First email notification of excessive slowness issues from districts sent to firstname.lastname@example.org.
At 8:06 am - Eduphoria Support Team notified Technical and Infrastructure teams to determine cause.
At 9:20 am - Eduphoria Infrastructure Team determined source of a performance issue in Forethought.
At 9:45 am - Eduphoria Infrastructure Team prepared a build for release of performance improvement in Forethought.
At 9:46 am - Eduphoria Technical Team determined source of a performance issue in Aware. Team is working to correct performance.
At 10:00am - Eduphoria Support Team received continued reports of saving and performance issues in Forethought.
At 10:10am - Eduphoria Infrastructure Team prepared a build for release of performance improvement in all systems.
At 11:00am - Eduphoria Infrastructure Team released an update including performance improvements in all systems.
At 11:13am performance appears to be fully restored.
Details of solutions:
Several issues were detected over the past 10 days, through 2018-10-15, that impacted system performance. Windows Server 2016 was failing to properly terminate TLS/SSL connections after a recent Windows update. Improvements included moving TLS termination to the load balancer.
Through tooling we discovered several performance issues, separate from TLS termination, that could be improved. Our team has resolved several of these items and will continue to release updates to improve performance.
Last night we repaired one final issue that was designed to reduce calls to a specific service that was not necessary for pages to render. This change impacted Forethought loading and saving plans. We rolled the release back and deployed a corrected version.