Microsoft’s Automated Traffic Shift Triggered a Global Service Outage
Microsoft experienced a global service outage last week, which the company attributed to a latent system flaw. [[fact:fact-6k165659a]] A preliminary report Microsoft sent to customers said the flaw had been present in its data centers since July and had gone undetected. [[fact:fact-wjq5t0i9c]] On November 11, an automated traffic shift moved Teams meeting and instant-messaging traffic to another server group that also handled Outlook traffic. [[fact:fact-pscm81v5c]] Microsoft’s automated failover system then moved traffic and email to servers it deemed healthy, carrying the same flaw into those systems and triggering a cascade of failures. [[fact:fact-7bdgkkzd0]] At the outage’s peak, more than 80% of Outlook backend servers were unresponsive. [[fact:fact-udspv4bhe]] It took about 15 hours from scattered failures for Microsoft to confirm the full scope of the problem, and roughly another 15 hours before it began deploying a fix. [[fact:fact-ys2givv9m]]