Event Start Time: 6:15 am
Event End Time: 7:45 am
Issue: Front end web servers did not autoscale correctly due to an adjustment for daylight savings time.
At 6:15 am - First email notification of access issues from customer sent in to email@example.com.
At 7:16 am - Eduphoria Support Team notified Technical and Infrastructure teams to determine cause.
At 7:20 am - Eduphoria Infrastructure Team determined cause was due to delay in auto scaling as a result of daylight savings time shift. Override was performed to adjust front end server pool.
At 7:38 am - Eduphoria Infrastructure Team confirmed all web servers had scaled up to the proper level.
Event Start Time: 8:30 am
Event End Time: 10:13 am
Issue: Database write servers experienced high load.
At 8:30 am database servers were experiencing heavy load. Due to manual modification of the auto scaling tools, an automatic update process started and added heavy load to several databases.
At 9:15 am the Eduphoria Infrastructure Team removed the automatic update process impeding optimal performance.
At 9:25 am the Eduphoria Infrastructure Team began clean up additional processes that were already running and impeding optimal performance on individual servers.
At 10:13 am the Eduphoria Infrastructure Team confirmed all databases had reached optimal performance.