Software Bug Dating Back to July Caused Massive Global Microsoft 365 Outage, Microsoft Reveals

More than 80% of the network gateway servers supporting Exchange Online were unresponsive at the worst point of the 12 November outage, which severely disrupted corporate and government customers’ day-to-day operations worldwide. The episode again exposed the operational weaknesses and concentration risks arising from heavy reliance on one cloud provider. On Monday, a Microsoft executive vice president issued a formal apology on X, after a preliminary report for customers had appeared on Sunday. Faulty gateways continued to serve Outlook and Exchange Online during the busiest US business hours, whereas applications including Teams, OneDrive and SharePoint promptly switched to functioning infrastructure.

TMGM Analysis: Financial Market News, Economic Calendar & Market Insights

Three operational factors together brought about the disruption. The primary network gateway was already processing Outlook traffic when routine automated traffic management diverted Teams traffic to it late on 11 November. Its available server capacity had also been reduced by approximately 14% during unrelated maintenance the day before. With that capacity offline, the system had no headroom to absorb the additional traffic.

Servers began exhausting their memory and failing in sequence on 12 November, after the traffic surge activated an internal lock contention bug embedded in a July gateway update. Explaining why standard pre-deployment laboratory tests had missed the fault over the previous three months, Ramanna said the testing environment had given priority to older hardware with fewer processor cores. Newer processors with higher core counts were more susceptible to the contention issue.

Rather than relieving the pressure, automated efforts to send users to functioning nodes overwhelmed the servers still operating and caused the outage to spread faster. Regional containment also failed: a separate issue prevented the standard partition isolation mechanism from confining the disruption to particular regions. The bug therefore reached Microsoft’s global server infrastructure in less than two hours.

Approximately 15 hours were needed to establish the root cause, followed by another 15 hours before engineers started deploying a fix, making the recovery unusually lengthy. Standard rollback tools and feature switches offered no effective remedy, as the defective code had already been running for months. Recovery required engineers to disconnect more than half of each affected region’s infrastructure at a time, reverse the code changes progressively and reintroduce the backlog of user traffic in stages.

Market Insight: Cloud Reliance Complicates Infrastructure Management

As companies rely more heavily on integrated productivity tools and AI services, the task of managing hyperscale cloud infrastructure is becoming more complex, a challenge underlined by this incident. Closer scrutiny of Microsoft’s load testing procedures and failover safeguards is likely from institutional customers and market observers. Their focus will be on ensuring that high-density hardware deployments do not introduce concealed systemic risks.