Three small failures combined to take down Microsoft 365
Microsoft’s preliminary report attributed a broad outage affecting Microsoft 365 and Exchange infrastructure to a latent cloud gateway defect, traffic changes, and reduced server capacity. [[fact:fact-brkqnfip9]] Microsoft said a cloud gateway change deployed starting in July introduced a defect that could exhaust CPU and memory resources and become unresponsive under certain conditions. [[fact:fact-lh402bk3w]] Microsoft said pre-release testing missed the defect because the test hardware was less powerful than some systems already deployed in production. [[fact:fact-qe997jn9e]] Traffic shifts during standard network maintenance and upgrades exposed the latent defect. [[fact:fact-vchk52pm4]] About 15 hours before the outage, Teams traffic was moved from the alternate gateway to the primary gateway that maintains Exchange Online access. [[fact:fact-tq90yhvyq]] Affected servers then exhausted capacity and refused new connection requests as requests continued to pile up. [[fact:fact-ttjzez2el]]