OnBoard Incident Report
Incident Date: 7/20/2017 7:30 PM EST
Incident Description: UK service outage and degradation in other regions.
At 7:30PM EST, we were notified of slow response times to the system. Upon immediate investigation, Azure was reporting that a network appliance caused an outage in the UK region across all Azure customers. This was causing the US server requests to wait for a SQL timeout before returning all organizations except data located in the UK. Since the data from the request was known to be incomplete, the cross-region request was not cached. This caused initial page requests to take upwards of 2-minutes for the SQL timeout to be achieved. All UK resources were immediately unavailable at that time.
As a temporary mitigation for non-affected data by the outage, we pulled all UK servers/resources from the pool of availability within the OnBoard system. This allowed us to bring full stability back to the US and Canadian regions. We had the service degradation to those two locations mitigated by 7:50 PM EST, however, the UK region remained down until Azure staff could repair the networking appliance.
Azure reported service was restored around 10:00 PM EST to the UK region, and at that time, we restored UK resources back into the general pool of availability to OnBoard. The system was actively monitored for other issues for the following hour and found no other issues to be handled at that time.
Please sign in to leave a comment.