Rendered at 03:07:37 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
pwthornton 1 hours ago [-]
Nadella stripped mined the company for AI jank, and everything has been falling apart. Github has been unreliable a lot lately. Xbox is basically imploding. A lot of Azure, Entra, Intune, etc. is shockingly janky, and there is clearly very little investment in it.
llama052 1 hours ago [-]
I’d imagine they are going to get a massive drain of whatever talent is left from their company wide severance as well.
We use Azure at work and it’s wild how much stuff is just manual labor behind the scenes on their platform.
Seems like it’s always on fire somewhere, ripe for burnout and MBOs aren’t aligned with reality from what I hear.
rsalus 5 minutes ago [-]
There are huge swaths of azure engineering teams that deploy things entirely manually via azure portal. Insane.
senderista 4 minutes ago [-]
> We use Azure at work and it’s wild how much stuff is just manual labor behind the scenes on their platform.
AWS is no different. Everything is on fire all the time, but customers usually don't notice.
juujian 39 minutes ago [-]
Hadn't heard about Xbox, what's happening?
OnlineTux 28 minutes ago [-]
Massive layoffs and studio closures and mergers like Obsidian merged with Bethesda.
New Xbox to cost $1200-$1800.
Gamepass limited to 15 hours a month for Ultimate.
12_throw_away 51 minutes ago [-]
I don't know exactly what's going on with today's outage.
But this is certainly exactly what I'd expect after indiscriminately laying off half your workers, then immediately offering a buyout to make sure any experienced people still standing will find their way to the door.
If this incident, or the difficulty they're having with remediation (update: holy shit, they still haven't even identified the problem), are somehow not caused by any of that, then it's at least a preview of what things are going to be like.
kstrauser 17 minutes ago [-]
OTOH, I couldn’t manage Entra without their AI chatbot. That system is a Lovecraftian stack of sadness built upon despair, of which mortals are not meant to know.
nullwarp 3 minutes ago [-]
This really is the honest truth, every interaction I have with the MS platform is somehow worse than an actual dumpster fire. AI is the only thing keeping the ship afloat. Then again maybe that's the point?
AIorNot 44 minutes ago [-]
So a few years ago Nadella was a God among CEOs capturing OpenAI at the right time etc..but yeah laying off thousands of tenured folks surely is great strategy, god I hate the tech industry.
Btw does everyone see how insanely crappy Windows 11 is compared to MacOS
fractal618 12 minutes ago [-]
I made the switch to Linux Mint and it’s wonderful and almost triples my battery life. I’m also in the process of switching to a phone with graphene os which runs like butter and feels so much more secure, safe and private.
jwilcoxson 36 minutes ago [-]
Yup.
beowulfey 7 hours ago [-]
Copied this from my post in the other thread:
The Status page is not working well, but I managed to subscribe to their RSS feed and it has been sending out updates periodically. I'll post some of them here.
9:02 PM UTC: We've validated an alternative health check method on a subset of affected infrastructure, which allowed the servers in that subset to successfully return to service, and we're now applying this change more broadly. We're also preparing an automated recovery process to continually detect and restart unresponsive servers, to help keep them in service while this change progresses. Additionally, our review of recent changes has identified a small number of configuration updates that we're investigating as potential contributors to the issue.
Previous status: While validating the targeted change, we found that many of the affected servers have become unresponsive, which is preventing us from confirming whether the change is effective. We're restarting these servers so they can come back online with the change in place, and we're applying the same approach to additional affected portions of our infrastructure to validate the results more broadly.
8:52 PM UTC: While validating the targeted change, we found that many of the affected servers have become unresponsive, which is preventing us from confirming whether the change is effective. We're restarting these servers so they can come back online with the change in place, and we're applying the same approach to additional affected portions of our infrastructure to validate the results more broadly.
Previous Status: While we haven't identified any recent changes that correlate with the start of impact, our investigation indicates the failed health checks may originate from a specific component on the affected servers that's responsible for processing them. We're deploying a targeted change to bypass this component on a subset of servers to validate whether this restores availability, and we've taken steps to prevent additional servers from being removed from service while we work to return the affected servers to service. We're also investigating why affected servers are experiencing elevated processor usage while handling requests.
7:52 PM UTC: We've identified that affected servers are failing automated health checks, which is causing them to become unavailable, and this may be related to a recent change to a component involved in processing those checks. We're testing a configuration change on a subset of affected infrastructure to address this and, if successful, will apply it more broadly. In parallel, we've initiated a restart of all affected servers and are validating whether they're successfully returning to service.
7:30 PM UTC: We're aggressively pursuing all pathways to reduce impact. As a precaution, we've paused all deployments and configuration changes while we review recent changes for any correlation to this event. We've determined that the elevated load isn't driven by an overall increase in traffic, but by a large number of servers being unexpectedly unavailable, which is shifting load onto the remaining infrastructure. We're investigating what's causing these servers to become unavailable while we work to safely return them to service.
7:16 PM UTC: We're aggressively pursuing all pathways to reduce impact. As a precaution, we've paused all deployments and configuration changes while we review recent changes for any correlation to this event. We've also moved traffic for some services off the affected infrastructure to further reduce load, with additional traffic moves in progress.
6:59 PM UTC: We've completed redistributing connection traffic across our worldwide infrastructure, and we're moving additional traffic off the affected infrastructure to further reduce load. We're also restarting affected servers to return them to service. In parallel, we're investigating whether the elevated traffic reflects a genuine increase in requests or a shift in how traffic is being routed across our infrastructure.
no-dr-onboard 6 hours ago [-]
This is like getting transmissions from Cassini before it falls into saturn.
hammock 5 hours ago [-]
Why is there not a single mention that this could be a cyberattack? Is the issue only affecting the USA?
outworlder 4 hours ago [-]
Because there's no indications that it is the case, at least for the time being.
testrun 3 hours ago [-]
Australia affected as well.
lazyasciiart 4 hours ago [-]
No, my family is in Europe and Australia and also seeing it.
skygazer 21 minutes ago [-]
I can connect to US account services, but while first VPNing through France, rather than directly from US.
Prickle 3 hours ago [-]
Japan is also seeing the issue. It is probably a global issue.
colechristensen 3 hours ago [-]
It doesn't look like a cyberattack at all, talking about fixing health checks and the like looks much more like broken automation.
King-Aaron 2 hours ago [-]
Impacting us here in Australia, our company is having fun today.
gosub100 1 hours ago [-]
Don't attribute to malice that which can be explained by stupidity.
Exmoor 1 hours ago [-]
> Don't attribute to malice that which can be explained by DNS.
Perhaps more accurate in our world.
elmo2you 4 hours ago [-]
Nope. Europe (eastern) for sure included too.
Angostura 3 hours ago [-]
UK seeing it
evandrofisico 2 hours ago [-]
Brazil too
xarope 1 hours ago [-]
I work for a global company, it's affecting every single of our 50+ offices world through US, Europe, Asia, Middle East, etc.
qsfp28 5 hours ago [-]
[dead]
crunkmcgoop 4 hours ago [-]
[dead]
razingeden 13 minutes ago [-]
Cortana, play My Heart Will Go - oh.
EvanAnderson 6 hours ago [-]
I feel bad for the people working on this. It sounds like desperation is increasing.
From the 10:11:39 PM UTC update:
Current status: We're continuing to pursue all available pathways to restore service, as our actions taken thus far have not provided the anticipated relief. Our investigation has found that a core process on the affected servers is becoming overwhelmed, which causes the servers to stop responding and contributes to the capacity loss.
To validate a potential recovery approach, we're fully isolating a portion of our service infrastructure, restarting the affected components, and gradually reintroducing traffic to confirm our services remain stable before applying this approach more broadly. We're also analyzing traffic patterns across our entire service infrastructure and evaluating whether isolating traffic within each region may allow affected regions to recover independently.
From the 9:41:46 PM UTC update:
Current status: We're continuing to work through restarting and validating the affected servers to safely return them to service, and we're exploring opportunities to expedite this process. In parallel, we're assessing whether reverting the small number of recent configuration updates we identified could help accelerate recovery.
From the 9:01:14 PM UTC update:
Current status: We've validated an alternative health check method on a subset of affected infrastructure, which allowed the servers in that subset to successfully return to service, and we're now applying this change more broadly. We're also preparing an automated recovery process to continually detect and restart unresponsive servers, to help keep them in service while this change progresses. Additionally, our review of recent changes has identified a small number of configuration updates that we're investigating as potential contributors to the issue.
From the 8:44 PM UTC update:
Current status: While validating the targeted change, we found that many of the affected servers have become unresponsive, which is preventing us from confirming whether the change is effective. We're restarting these servers so they can come back online with the change in place, and we're applying the same approach to additional affected portions of our infrastructure to validate the results more broadly.
Previous Status: While we haven't identified any recent changes that correlate with the start of impact, our investigation indicates the failed health checks may originate from a specific component on the affected servers that's responsible for processing them. We're deploying a targeted change to bypass this component on a subset of servers to validate whether this restores availability, and we've taken steps to prevent additional servers from being removed from service while we work to return the affected servers to service. We're also investigating why affected servers are experiencing elevated processor usage while handling requests.
NolF 46 minutes ago [-]
This seems like the same terse language/feedback OpenAI models give me and I leave more clueless than I arrived.
EvanAnderson 4 hours ago [-]
Working in reverse-chronological order here:
From the 1:06:17 AM UTC update:
Current status: Our investigation has found that a process on the affected servers responsible for establishing secure connections becomes overwhelmed, causing the servers to stop processing requests. We're preparing a configuration change to address this behavior that can be applied across all affected servers, and we're validating it on a subset of servers before applying it more broadly alongside restarts of the affected infrastructure. In parallel, we've begun reverting a recent network configuration change that correlates with the start of impact and are monitoring to determine whether this contributes to recovery.
The temporary traffic limitations we previously described are now in effect in specific locations. We're monitoring the affected servers in those locations to confirm whether they remain stable while handling a narrower set of connections, after which we'll gradually reintroduce additional traffic.
---
The list of impacted services just grew tremendously.
From the 12:56:09 AM UTC update:
Features of the Microsoft 365 services below that interact with Exchange Online and email services are impacted. Elements of those services that do not depend on interaction with Exchange Online remain functional.
- Exchange Online: Users may be unable to access their mailbox using Outlook on the web, Outlook desktop (MAPI), Exchange ActiveSync (EAS), or REST-based connections, and email delivery may be delayed.
- Microsoft 365 admin center: Admins may be unable to load some admin center pages.
- Microsoft 365 apps: Users may be unable to view their recently used files.
- Microsoft 365 Apps Admin Center: Multiple services including Cloud Update, Cloud Policy, and Inventory are affected.
- Microsoft 365 Education: Features including Assignments, Reading Progress, and Speaker Progress may be impacted.
- Microsoft Copilot 365: Users may be unable to load Copilot Chat or receive responses in the web and desktop apps, and Copilot experiences may be unable to access work data such as email, calendar, and files.
- Microsoft Defender for Office 365: Safe Links may be unable to verify URLs, which may impact users clicking links in Outlook and Teams.
- Microsoft Defender XDR: Users may be unable to load Microsoft Defender portal pages or solution tabs.
- Microsoft Fabric: Users may encounter failures when exporting reports with sensitivity labels to PDF or PowerPoint, or when updating artifacts with sensitivity labels.
- Microsoft Graph: API requests that access Exchange Online data, such as mail, calendar, and contacts, may time out.
- Microsoft Intune
- Microsoft Planner: Users are experiencing Plan and Task operation reliability issues.
- Microsoft Purview: Multiple Purview services including Authorization, auto-labeling, Data Loss Prevention (DLP), Communication Compliance, and Data Classification are impacted, and users may be unable to load Microsoft Purview portal pages.
- Microsoft Teams: Scenarios that depend on Exchange Online, including calendar, meeting scheduling, and creating chats for organizations using Information Barriers, may fail. Core messaging and online meetings remain largely functional.
- Microsoft To Do: The app may not load, and users may see operation failures.
- Power BI: All users may intermittently be unable to export reports with MIP sensitivity labels to PDF in Power BI.
- SharePoint Online: Features that depend on Exchange Online may be impacted. Accessing files stored directly in SharePoint Online remains functional.
- Universal Print: Users may encounter errors submitting print jobs or using admin features.
- Windows 365: Customers who have configured their tenant and Cloud PCs to use Microsoft Purview Customer Key for Windows 365 Cloud PCs may have users who are unable to connect to their Cloud PCs.
Current status: We've reviewed known impact scenarios and have revised the list of impact services and features, which can be viewed in the "more info" section of this communication.
We're continuing our work to isolate the issue affecting server availability, as well as other workstreams.
---
So much for seeming hopeful.
From the 12:08:53 AM UTC update:
Current status: While the servers we've returned to service initially remain healthy, we've observed that they can become unresponsive again once they begin receiving traffic. We're analyzing diagnostic data from these servers with our networking and platform experts to determine what's triggering this behavior.
Building on the protective traffic management measures we've implemented, we're also temporarily limiting which types of traffic can reach the affected infrastructure in specific locations. This will help us determine whether a particular type of request is contributing to the impact and allow us to restore service incrementally. Additionally, we're rebuilding a subset of affected servers using a previous software configuration to validate whether this prevents them from becoming unresponsive, and we'll expand this approach to additional servers if we observe positive results.
---
The 11:05:31 PM UTC update is is the first since the outage began that feels hopeful.
From the 11:06:31 PM UTC update:
Current status: We've begun returning a portion of the affected servers to service, and they're remaining stable so far. We've also shifted traffic and connections to alternative pathways wherever possible and have observed limited recovery in some regions.
We've streamlined our recovery process to expedite returning the remaining affected servers to service. To further support recovery, we're implementing protective traffic management measures in a specific location to reduce load on the affected infrastructure, and we'll expand these measures to additional locations if we observe positive results. As capacity is restored, users may experience a gradual recovery as their clients reconnect to the service.
EvanAnderson 1 hours ago [-]
Hopefulness continues, I guess?
From the 2:14:51 AM UTC update:
Current status: We're applying infrastructure capacity improvements where possible in an effort to help support traffic in regions coming online. We're continuing to monitor the results of changes applied so we can identify strategic actions that can help to improve service health across all affected regions.
---
Hopefulness returns.
From the 1:50:27 AM UTC update:
Current status: We're seeing positive results with our efforts to restart affected infrastructure. We're actively monitoring the performance of these to verify stability which will support improved availability. Remediation actions will be expanded as we confirm stability with the changes applied.
matt3210 4 hours ago [-]
Sounds like nobody knows how their own systems work anymore.
hn_throwaway_99 3 hours ago [-]
While I think it's a bit premature to jump to "AI is at fault", and I will emphasize that I appreciate Microsoft's transparency in their status updates, these updates do have much more of a "we don't know what's going on" feel than any other major outage I can think of.
It's been hours, and the status updates still feel like there is an undertone of frantic panic (understood if you can't determine the root cause). When I think of past major outages, they may have been down for hours but usually folks knew what the root cause was relatively early, even if remediation took some time.
I wouldn't be surprised if AI coding and release pressure was a primary cause, so I really hope the future post-mortem is made publicly available.
2 hours ago [-]
riffic 2 hours ago [-]
Nobody has ever fully understood how their systems work. Thinking largely about "systems as imagined" and "systems as found."
Richard Cook wrote about this years ago, and it's always going to be relevant:
Nah, they're just debugging a hard problem while the world looks on! The 'normal' problems they have processes to work through, but hey emergent systems and all that - they have to expect the unexpected.
fock 5 hours ago [-]
their employer brought me agent swarms supposed to take over my php-serving machines. whatever this is, I don't feel sorry for them.
piker 4 hours ago [-]
So the thundering herd wins again
reaperducer 4 hours ago [-]
Ugh. There's some yahoo from NewsNation on WGN Radio right now saying this is because of the Chinese, and obliquely linking it to Nvidia.
Reddit-level speculation.
llama052 4 hours ago [-]
My girlfriends work she is hearing this from office employees. Kind of scary how much the public gets wrong.
eddythompson80 5 hours ago [-]
they are furiously typing at their agents hoping they can figure out what to do. Getting glimps of hope everytime claude says "I found it." only for their hopes to be crused at "Turns out ...". The clown show at Microsoft is surreal.
Rantenki 4 hours ago [-]
I suspect that they are not. Copilot is also down :O
5 hours ago [-]
5 hours ago [-]
4 hours ago [-]
wmchen 7 hours ago [-]
Not really a big deal, but it is presently 8pm UTC time. I'm curious why their RSS feed seems to be 2 hours off?
beowulfey 7 hours ago [-]
Ah no that was a mistake on my part! RSS Feed was in EST and I misremembered the difference
fock 7 hours ago [-]
[flagged]
7 hours ago [-]
SubiculumCode 2 hours ago [-]
Gonna have to rebrand as Office364
jwilcoxson 34 minutes ago [-]
Well done sir
jiggawatts 9 minutes ago [-]
We already call it Microsoft 356.
Eji1700 4 hours ago [-]
Day 12: It has been 3 days since the last ping returned, I fear I will not make it back to my family. Give my love to dear Liza.
I feel for those stuck in there. The outage going on for this long has got to be fucking brutal.
dudus 7 hours ago [-]
So the servers are ok but many of the health checks are failing and so the servers are taken off rotation, increasing load on other servers. I'm sure it's more complicated than it seems but from my POV this looks like amateur hour at Microsoft.
shermantanktop 5 hours ago [-]
Amateur hour? Not sure if you've lived through one of these cascading failures, but they are hard to design against, hard to load/chaos test for, and hard to recover from. The key is that this is a dynamic load, not a static load; and it's normally amplified by uncontrolled retry behavior by clients.
In retrospect, the cause and cure are obvious to everyone.
UltraSane 1 minutes ago [-]
The best defense against cascading failures is limiting the blast radius, which is something AWS seems to do well but Azure does not.
YZF 2 hours ago [-]
I have lived through and debugged cascading failures and large scale distributed systems.
The problem is amateur hour is the entire industry. Nobody specs anything, nobody tests overload conditions, nobody thinks failure modes. The stuff that every engineer does when building anything else other than software doesn't happen. The orgs are bloated. Too many teams. Too many silos. Incentives are stacked the wrong way. People are motivated to not think about systems but stay in their lane. Responsibilities are split in weird ways.
Yes. Big systems are hard. But we can build, manage and understand big systems with the right approach and people. Even one computer is complicated and many people get stuck trying to debug a piece of software running on one machine. At the end of the day it's not magic. All of humanity is big systems but when I have a power failure the power company is generally able to understand why and resolve it and the water in my tap doesn't stop flowing and the cell towers still work (generators, who would have thunk).
A friend who is really good had to leave Microsoft because of how AI was pushed. Before that he was telling me some other stories that make the culture seem pretty not great. Heard similar stories from other people. It used to be a great place filled with great people until at some point it's just another big company.
torben-friis 12 minutes ago [-]
I'll be the downvoted comment, since HN doesn't like to hear this, but the other flavors of engineering do stuff we don't because they're regulated.
My bet is we'll eventually get there. Planes kill people when they fall, software used to not be as dangerous, but it's been a while since it became critical infrastructure.
mschuster91 4 minutes ago [-]
> My bet is we'll eventually get there.
Given that people on reddit claimed that in the entirety of New Zealand no real estate transactions can be settled, causing cascading failures [1]...
I think this might actually be the incident pushing governments to designate large online services such as MS 365, Gmail et al as critical infrastructure and force companies to massive, yet-unseen levels of transparency and regulation.
The EU is already itching to act ever since the Trump admin forced US Big Tech to terminate service to the ICC judges, and this here was a square own goal for Microsoft.
Except for the fact that it seems like most Microsoft outages are self owns. I'm not sure how many times we've had outages due to bad deployments that somehow made it to their global infrastructure and only got noticed when customers notified.
Plenty of internal accounts from previous employees highlight this.
motionlessveloc 39 minutes ago [-]
To be fair to Microsoft (and I'm having trouble believing I just wrote that), the vast majority of outages are "self owns". If you don't change anything, you're not going to break anything, and major external events (fires, cable cuts etc) are rare and can usually be worked around.
infamouscow 4 hours ago [-]
Many of us insist upon not building Rube Goldberg machines that cannot be reasoned about precisely for this reason.
Most of the time we get demonized by resume-driven developers job-hopping every few years (seems like the lack of deep expertise is working out). Today we get to laugh at their expense. These problems were entirely foreseeable and avoidable.
lebuffon 2 hours ago [-]
Isn't a complex computer system, at the lowest levels, an example of a Rube Goldberg machine ?
If we could see the memory bits and the CPU internals, I think it would be good candidate, particularly branching and branch prediction.
YZF 2 hours ago [-]
There are methods for building complex systems our of smaller pieces. Humanity has done this forever. An internal combustion engine is very complex. An airplane is very complex. You put a boundary around your system, you build it to a spec, you test it to a spec, you define the behaviour across anything you can see. When two engines fail on the airplane it can maybe still at least glide. Maybe there's an ejection seat. Because someone is doing the system thinking with some level of discipline.
A brick is very complex. A bearing is complex. A bolt and a nut and a washer are complex. An integrated circuit is complex. We build stuff out of that. Like buildings that don't fall when you sneeze on them.
In software we use duct tape - at best. But that's not just Microsoft, that's pretty much everyone. With maybe few exceptions.
AIorNot 40 minutes ago [-]
Software systems are by nature more complex than physical systems. They also lack any tangible interfaces beyond code and diagrams and collective knowledge in the minds of the humans who build and maintain them.. The rapid evolution of software is no help either..if anything AI is accelerating that rapidity as well.
EFreethought 59 minutes ago [-]
In my experience it is always the "business" people that do not want to take the time to do it correctly.
Den_VR 4 hours ago [-]
> building Rube Goldberg machines that cannot be reasoned about
What a great phrase!, can I use that in conversation?
mminer237 4 hours ago [-]
Newest update says that wasn't the issue after all. They say relatively small loads are actually causing servers to become unresponsive and have to be restarted.
They still don't know why this "core process" is maxing out the processor for no reason.
SoftTalker 51 minutes ago [-]
I'd start with "what has changed" but surely they've looked at that.
burningChrome 4 hours ago [-]
This sounds suspiciously what occurred when I was working at a large motor sports company. We had a major release of new year models coming out on the website. The deployment went super smooth and then in about 30 mins the site was crashing. Since we were a huge MS client, almost everything we ran was all MS top to bottom.
Site was taken offline, war rooms ensued. 48 hours later, an Ops team from Redmond was flown out. Another three days and they determined it was several legacy servers were going down and the load balancing servers were getting hit so hard, they also failed. It was a cascading effect that killed everything.
CIO was pissed. The company got absolutely lit up on every social media channel for their ineptitude. The CIO vowed it would never happen again, they were upgrading the entire network and infrastructure to avoid it happening again. Fast forward six months to the next major release. Same thing, same panic, same cascading failures because servers were collapsing under the traffic to the website. Another war room, another Ops team flies out. Two days later they figured out it was some buggy code that was causing it.
This after 8 rounds of load balancing testing to the tune of 10X the amount of traffic the site sees on these major release dates without so much as a hiccup.
I left two months later, but the overall feeling was the same - total amateur hour from Microsoft after our company was gifted major discounts to go all in with their hardware & network gear. The company found out the hard way it wasn't a smart decision.
The scary part was this some 20 years ago - seems like not much has changed considering how frequently Github is down now too.
semiquaver 6 hours ago [-]
The people writing these updates are very obviously not technical and are least one step removed from the actual work being done to fix this.
EvanAnderson 5 hours ago [-]
To be fair: I want the people who can speak most competently to a problem to be working on it versus writing commentary. Those highly-technical people can shine in the postmortem report.
skeeter2020 4 hours ago [-]
I disagree; the communication aspects - including external - are a hugely important part of any incident response. IME the biggest challenges in these situations are the many levels & combinations of communication and coordination. You don't need your best DevOps person doing it, but there should be plenty of competent technical product managers or engineering managers who could do this well.
codingdave 4 hours ago [-]
And that is a good thing. Well, maybe not good. But the last thing you need in a crisis is to pull people away from fixing the problem and make them type up status updates. Admittedly, having tech knowledge would help the writers - a technical incident management team would be better. But ain't nothing wrong with letting the PR folks write up words while the tech folk work on problem solving.
Actual tech write-ups can come later.
6031769 7 hours ago [-]
At Microsoft every hour is amateur hour.
mapontosevenths 6 hours ago [-]
I'm not thrilled with what they've done to Windows lately myself, but this is just rude.
You know a lot of talented people who work there also post here, and it's not fair to insinuate that they aren't talented because some vastly complex system somewhere in the world had a bad day.
Stuff happens, things break. That's why lots of nines cost a lot of money. It doesn't automatically mean that anyone is incompetent.
eddythompson80 4 hours ago [-]
Individual talent is irrelevant when you have a company culture that rewards the opposite of that. Any talented person working at Microsoft knows that they are dealing with structural problem that force MS products to be massive piles of shit. There isn't much an talented individual can do about that.
hyperhello 2 hours ago [-]
An organization can become fundamentally opposed to the purpose for which it was created. Organizations simply don't follow the same laws as the individuals inside it do. A company can be made of geniuses and be totally ineffective; it can be full of talented go-getters and be completely unable to fix problems.
buttslut5099 42 minutes ago [-]
Satya sock account found
fyrabanks 5 hours ago [-]
i agree with everything you said here except the last part.
lots of nines means not having significant downtime. we passed 99.99% hours ago, and we're on the way to 99.9%
codingdave 3 hours ago [-]
Lots of nines means they have SLAs of no significant downtime. It does not guarantee that the SLAs can never be missed, just that there will be consequences when it is missed. Lawyers and accountants and sales folks will sort out the compensation payments that come back to the customers and try to negotiate relationships back to a healthy state.
ifwinterco 4 hours ago [-]
Stuff happens and things break everywhere but right now at Microsoft they break badly and often, that points to actual issues.
Almost certainly not the fault of individual people, it's a management failure of course
imglorp 39 minutes ago [-]
The enshitification of Windows was not technical incompetence; it was intentional product direction.
They transformed Windows from a useful OS to a banal surveillance and advertising platform.
TheOtherHobbes 12 minutes ago [-]
Microsoft's superpower has always been hiring smart people and making them work under dumb management to create flimsy, mediocre outcomes.
zmgsabst 4 hours ago [-]
There are lots of talented amateurs. What separates professionals is standards, quality, teamwork, etc.
I agree there are many talented people at Microsoft, but I don’t see the professionalism in this incident.
bradleyjg 6 hours ago [-]
Don’t worry, I’m sure it’ll get better after they layoff 60,000 people.
ahartmetz 4 hours ago [-]
Don't worry, their replacements will receive very clear instructions instructing them to make no mistakes.
6 hours ago [-]
NoPicklez 3 hours ago [-]
I can't stand it when I see armchair analysts making claims like this.
I'm sure it's more complicated than that.... but from my POV, which is extremely limited and informed by what others online tell me.... that this is amateur hour
What would it take for it to be professional hour?
TheOtherHobbes 8 minutes ago [-]
Professional hour would be invisible. Outages don't happen because they were anticipated and prevented.
You can't do much to work around nuclear war or a major asteroid strike, but keeping important things working under normal operating conditions should not be a high bar.
steve1977 6 hours ago [-]
Copilot hour!
ArcHound 7 hours ago [-]
They are turning it off and on again. I don't even. If this works, our industry is cooked as the kids say.
onefiftymike 6 hours ago [-]
I worked on multi-million dollar airplanes. If a pilot called me with an issue and hadn't rebooted the plane (on the ground), I wouldn't even talk to them. Rebooting isn't always a bad solution to technology.
ArcHound 6 hours ago [-]
It's not a bad solution, it definitely should be tried. But I'm still rather shocked to see it in official notes, especially this "late" in the timeline.
mapontosevenths 6 hours ago [-]
It's not a solution at all. It's a symptom.
3 hours ago [-]
olyjohn 1 hours ago [-]
If they rebooted it, and it started working again, would you just walk away? Or would you try to figure out why the fuck it needed to be rebooted in the first place?
SoftTalker 42 minutes ago [-]
Thales or Collins or Honeywell or whoever provided the system would. The technician at the airport would only follow a procedure listed on a checklist, which probably includes rebooting the system.
thisismyanon2 2 hours ago [-]
Rebooting is also the only solution for some problems.
looks at the Boeing 787
taftster 3 hours ago [-]
(on the ground) is probably pretty key, though. Ha ha. I can imagine the conversation if this was in the air.
justonceokay 7 hours ago [-]
In maintenance, sometimes the dumb solution is the best. The servers that ran amazon.com had a memory leak and were subsequently restarted. This hack continues for years because it was the most cost-effective thing to do.
Put another way, if they didn’t turn it off and on again, I would think less of them because that’s the first thing you should always do unless you know better
jimmaswell 6 hours ago [-]
I suppose you might want to first rule out a break-in, and if there is one, recover evidence that won't survive a reboot.
icameron 4 hours ago [-]
Powercycling is a designed method to bailout Cisco routers. If you botch a configuration command and lose connection, you can have someone powercycle it to restore you last good configuration. Its not just a joke it really works more often than I'd like to admit.
macintux 2 hours ago [-]
~25 years ago I was a firewall administrator, and the Cisco PIX we used was notorious for gradually running out of memory, effectively cutting off the entire company from the Internet when it went down (and it was a very big company).
We absolutely should have scheduled reboots during off hours. Lessons learned.
bawolff 6 hours ago [-]
Amateur hour. Everyone knows professionals say "power cycling" instead to make it sound complicated.
red-iron-pine 5 hours ago [-]
the layman understands both of those words and their meaning together.
we "bounce" the servers when on calls with customers.
martinald 7 hours ago [-]
Absolutely bizarre wording of these updates. Strangest corporate speak with urgency turned up to the max.
red-iron-pine 5 hours ago [-]
copilot, make this doom and gloom situation sound better. assume you're using the voice of a friendly but intensely focused sysadmin. use only neutral language.
jenimac926 7 hours ago [-]
They rolled back the 7:52 UTC update to the 730 one now....guess it didnt work
stevenhubertron 41 minutes ago [-]
What’s crazy is now hour 8 and my work email for 15k+ employees is still down. This is worse than the usual US-East-1 issues.
NoPicklez 38 minutes ago [-]
It's impacting all businesses running MS hosted Outlook, that would also include companies with 100,000 of employees. It's a global issues not just one area.
duxup 32 minutes ago [-]
US-East-1 is at least something you can kinda plan for / around. This isn’t.
goodluckchuck 14 minutes ago [-]
Yeah, with a brief outage, the concern is that you don’t have access. At this point, the concern is that someone else does.
soundworlds 17 minutes ago [-]
Yet HackerNews soldiers on.
tatersolid 4 hours ago [-]
My last email was received at 5.5 hours ago. Teams chat and meetings are the only thing in Microsoft 365 that seem to be working. All our Azure infrastructure seems fine so this is something 365-specific.
Everyone has occasional outages but this is the worst 365 outage our org has seen since starting with M365 in 2019.
pwarner 3 hours ago [-]
2018 featured a bad one.
Couldn't find a good article, but I remember it being basically all day.
I mean, I think the teams hate can be funny sometimes, but I honestly don’t get it. I’ve used all of the BIG programs. They’re all the fucking same. I can’t point to a single thing that Teams does drastically worse than any of the other options. I think it’s just a meme.
duxup 31 minutes ago [-]
I find the teams app to be really flakey. It changes time zones on me at random for a moment at a time.
doublepg23 1 hours ago [-]
I’d agree.
I think it does _too much_ and the performance is _bad_ but it’s rock-solid for video and audio calls - more than even FaceTime or analog calls.
conception 2 hours ago [-]
It has copied all the features it just does them all a bit worse.
jdironman 2 hours ago [-]
It's mainly that traumatic new message sound for me.
kstrauser 12 minutes ago [-]
Preface: I have zero love for Slack. It’s a thing. It’s fine. I can take or leave it.
Using Teams after being used to Slack is like going from VSCode to Eclipse. I’m sure this thing is powerful, but it looks weird, it feels weird, it moves like January molasses, and it has a million features I’m completely uninterested in. I’m sure some people get value from all the baroque. I am not one of them.
(I’d rather be using IRC and Emacs but the world has other ideas.)
SecretDreams 52 minutes ago [-]
> Teams chat and meetings are the only thing in Microsoft 365 that seem to be working.
Truly, we are in dark times.
soundworlds 6 hours ago [-]
Update from Microsoft:
"We've validated an alternative health check method on a subset of affected infrastructure, which allowed the servers in that subset to successfully return to service, and we're now applying this change more broadly.
"We're also preparing an automated recovery process to continually detect and restart unresponsive servers, to help keep them in service while this change progresses."
Update as of 10 minutes ago implies that didn't work:
> Current status: We’re continuing to pursue all available pathways to restore service, as our actions taken thus far have not provided the anticipated relief. Our investigation has found that a core process on the affected servers is becoming overwhelmed, which causes the servers to stop responding and contributes to the capacity loss.
cgio 4 hours ago [-]
So health checks were correct… this solution of silencing the first symptom is probably just adding to the problem. I am afraid they may be opening the door for the fire trucks and getting some other kind of intrusion instead. Better they shut it down for day and do proper fix.
EvanAnderson 5 hours ago [-]
The 12:08:53 AM UTC update is much less hopeful:
Current status: While the servers we've returned to service initially remain healthy, we've observed that they can become unresponsive again once they begin receiving traffic. We're analyzing diagnostic data from these servers with our networking and platform experts to determine what's triggering this behavior.
Building on the protective traffic management measures we've implemented, we're also temporarily limiting which types of traffic can reach the affected infrastructure in specific locations. This will help us determine whether a particular type of request is contributing to the impact and allow us to restore service incrementally. Additionally, we're rebuilding a subset of affected servers using a previous software configuration to validate whether this prevents them from becoming unresponsive, and we'll expand this approach to additional servers if we observe positive results.
---
The 11:05:31 PM UTC update is is the first since the outage began that feels hopeful.
From the 11:06:31 PM UTC update:
Current status: We've begun returning a portion of the affected servers to service, and they're remaining stable so far. We've also shifted traffic and connections to alternative pathways wherever possible and have observed limited recovery in some regions.
We've streamlined our recovery process to expedite returning the remaining affected servers to service. To further support recovery, we're implementing protective traffic management measures in a specific location to reduce load on the affected infrastructure, and we'll expand these measures to additional locations if we observe positive results. As capacity is restored, users may experience a gradual recovery as their clients reconnect to the service.
---
The 10:11:39 PM UTC update is a little chilling. They don't know why what's happening is happening so they're doing experiments.
I do that with closed-source software when I don't want to bother trying reverse engineer it, but these guys have the code! Yikes.
I recognize that even w/ the code available there are likely behaviors that aren't likely to be anticipated because of the scale of traffic. Complex systems fail in complex and non-intuitive ways.
From the 10:11:39 PM UTC update:
Current status: We're continuing to pursue all available pathways to restore service, as our actions taken thus far have not provided the anticipated relief. Our investigation has found that a core process on the affected servers is becoming overwhelmed, which causes the servers to stop responding and contributes to the capacity loss.
To validate a potential recovery approach, we're fully isolating a portion of our service infrastructure, restarting the affected components, and gradually reintroducing traffic to confirm our services remain stable before applying this approach more broadly. We're also analyzing traffic patterns across our entire service infrastructure and evaluating whether isolating traffic within each region may allow affected regions to recover independently.
ifwinterco 4 hours ago [-]
The code by itself doesn't tell you how the various components of a complex distributed system interact in a failure condition with clients hammering retries.
I mean it sort of does if you know the whole system inside and out and have the ability to reason about it, but just having the code for each component doesn't by itself solve the real problem of understanding
BLKNSLVR 5 hours ago [-]
> these guys have the code!
But maybe don't understand it because they didn't write it themselves?
LadyCailin 4 hours ago [-]
The one guy that did understand the code took VEO, and isn’t answering his phone right now.
we've observed that they can become unresponsive again once they begin receiving traffic.
Basically their server works well when they they are not used :-p
jp191919 5 hours ago [-]
Not exactly confidence inspiring.
5 hours ago [-]
k_sze 7 hours ago [-]
And I can't reach my self-hosted mail server in the Outlook desktop app because of this.
For reasons that I cannot fathom, Microsoft wants to act as the relay to my self-hosted e-mail account, if I add the account in Outlook.
jasonjayr 6 minutes ago [-]
Allegedly, It's so they can use push notifications efficiently from a central place for new mail notifications. Mobile IMAP clients cannot reliably keep a TCP connection open, due to on-device power-management and network management purposes. IMAP natively supports push-based notifications, but mobile networks make that impossible to work reliably.
Telemakhos 7 hours ago [-]
> For reasons that I cannot fathom, Microsoft wants to act as the relay to my self-hosted e-mail account
I can conjecture some reasons. Datamining would be high on the list, since MS can read your mail and thus market to you more effectively. Governments might also be happier if MS can provide them with one-stop access (like Prism) to otherwise self-hosted e-mail accounts proxied through a central server.
taftster 3 hours ago [-]
I would strongly doubt (though not completely rule out) that Microsoft is reading your personal mail hosted on your personal mail server. Like what would that even look like, their IMAP client is forwarding email to headquarters?
BUT, more likely, it's some sort of "check in" or "dial home" feature that isn't working. Or plausibly, maybe they count "local user checked email" and their analytics endpoint is down.
Regardless, it's definitely surprising that a traditional desktop application would be blocking when only polling from local/trusted data sources.
persolb 2 hours ago [-]
But a check in or analytic counter wouldn’t stop you from receiving email. This is most likely due to Microsoft’s server (not your local computer) being the one fetching the mail.
It is normally probably more reliable; except when their server is down.
basch 2 hours ago [-]
It probably is a mirror for caching purposes for the mobile app or something. It lets them control the protocol between the client and their server as well.
kspacewalk2 6 hours ago [-]
Or, you know, similar reasons as why people use Cloudflare as a reverse proxy.
rstupek 6 hours ago [-]
"New" Outlook does route all traffic through Microsoft servers. I assume it's so you can get a "unified" mail look on all your devices (phone, etc), but likely a nice source of data mining
lousken 4 hours ago [-]
outlook 2010 works just fine
ActionHank 7 hours ago [-]
This is your own fault for using their BS app.
ariamailclient 7 hours ago [-]
[flagged]
BLKNSLVR 4 hours ago [-]
Personally, I can't reconcile having a self-hosted email server and using Outlook to access said email.
I'm being an ass in saying this, because I'm aware human motivation is a beautiful kaleidoscope of variables, but I would have guessed at the venn diagram of 'those willing to compromise and use a Microsoft product' is mutually exclusive with that of the 'I'm hardcore enough to self-host email'.
I will say, however, that Outlook used to be one of the better desktop email programs, so I'll grant some legacy lenience.
drnick1 2 hours ago [-]
> Personally, I can't reconcile having a self-hosted email server and using Outlook to access said email.
In a corporate environment, I don't think this would be that unusual. But yes, it's hard to imagine someone would do that in a personal setup.
amanaplanacanal 4 hours ago [-]
What's your preferred email client? Just wondering what other people are using.
Mac/iOS Mail is pretty lightweight. I used to use Thunderbird on Linux.
prepend 3 hours ago [-]
Surprisingly, Mac mail is a really nice client. It reminds me of how quick eudora was back in the day.
sensanaty 3 hours ago [-]
Thunderbird still reigns supreme
SSLy 4 hours ago [-]
Best one I've ever used was Fastmail's web mail. But it's locked to their platform, so yeah.
drnick1 2 hours ago [-]
Thunderbird (desktop), K9 (Android).
BLKNSLVR 4 hours ago [-]
I use FairEmail on Android, which aggregates both my Gmail and self-hosted accounts.
I use Gmail's web interface for my Gmail account (which, I'll admit, makes my little rant above more than a little hypocritical)
And in my self hosted email setup I use RoundCube (web interface).
janfoeh 4 hours ago [-]
> (which, I'll admit, makes my little rant above more than a little hypocritical)
Na, that would simply be the kaleidoscope part you mentioned, wouldn't it? For what it's worth, I found the combination just as peculiar as you did.
zalebz 3 hours ago [-]
eM Client for various M365 and IMAP accounts.
ElectronMail for the 9 ProtonMail accounts I have (allows "tabs" for each account in one client).
andyfilms1 3 hours ago [-]
EmClient, Mailcraft
tantalor 2 hours ago [-]
Eudora
soupfordummies 11 minutes ago [-]
That’s a name I haven’t heard in a longgg time. Google says it’s still being used by some enthusiasts. Were you being sincere?
chneu 3 hours ago [-]
You're not being an ass. You're being a logical person who is advocating for individuals taking responsibility for their choices/actions.
Unfortunately, being someone who speaks up nowadays has been portrayed as being an asshole.
How dare you!?
jwilcoxson 31 minutes ago [-]
Why would you self host, then use a client with a dependency like this? Seems like the worst of both worlds.
ingaze 2 hours ago [-]
I’m pretty certain their desktop client applies filters server-side.
I’m pretty certain because my company’s phishing-test emails skip my filters (I assume admins have a “skip filters” toggle). Which, hilariously, makes it even easier to ensure there’s a place that contains only phishing test emails (filter all not on allow-list of domains to one place, filter all on that list to another, ta-da, the main inbox now contains only phishing test emails)
I assume they do the same thing with non-MS email if you use their client, rather than maintain two filter implementations.
All around, yeah, wow, super gross.
noAnswer 3 hours ago [-]
> For reasons that I cannot fathom
Biggest data heist and most obvious US intelligence operation in human history. I can't fathom why AVs don't treat "Outlook New" as info stealer malware.
brokenmachine 5 hours ago [-]
And I hope all of us have learned a valuable lesson.
emschwartz 7 hours ago [-]
I'm amazed to see so many services down and to see that Github and Github actions are still working
dingaling911 3 hours ago [-]
Every third build is failing for me failing to publish test results.
mook 7 hours ago [-]
Well, my mac CI runs aren't being picked up, and they eventually fail because of it. So maybe wherever connects the orchestrator to the machines is having issues.
emschwartz 7 hours ago [-]
I spoke too soon!
piker 6 hours ago [-]
So basically business as usual with the mac runners, right?
throooooo 3 hours ago [-]
I was gonna say. This happens to my Mac jobs on a normal basis.
shdjhdfh 4 hours ago [-]
GitHub infrastructure is for the most part completely separate from anything at Microsoft, other than using Azure.
dist-epoch 6 hours ago [-]
Round robin. Github Actions were down yesterday.
rvz 7 hours ago [-]
For now.
One of GitHub's services will be down again by next week.
sonicrocketman 9 hours ago [-]
Down detector shows like everything is down. AWS, Azure, Office, Gmail. Everything. https://downdetector.com
notatoad 8 hours ago [-]
down detector always looks like this during big cloud service outages. i think partly because people don't actually know what services they're using and report wrong things, and partly because downdetector parses social media posts like "outlook is down but gmail still works" as outlook and gmail both being down.
5 hours ago [-]
cwillu 7 hours ago [-]
Downdetector is a survey, not a detector, and suffers from user confusion when a high-profile service is actually down, as users start reporting unrelated things
kube-system 5 hours ago [-]
It also suffers from people reporting functionally related (but not technical related) things.
Any other service that interacts with Outlook in some way will create downstream effects in other services expecting Outlook to work normally.
hunter2_ 6 hours ago [-]
Especially when looking constitutes reporting (Schrödinger's downdetector?) and people look at everything after showing up for one thing.
midas89 7 hours ago [-]
I work in IT and people are complaining about Atlassian and about AWS but in reality if your organization is using SSO then that would get impacted.
If your org is using Intune or Defender EDR then you probably also are impacted.
jwnin 5 hours ago [-]
Entra is not impacted, SSO is working for non-M365 apps.
hydrox24 2 hours ago [-]
The biggest problem with downdetector is how they visualise their outage graphs.
The Y axes are entirely independent, and so when there's an outage for Microsoft Outlook (which peaked at ~130k reports) and you get a negligable spike in pepole who get confused and report gmail (~1k reports), the graphs all look correlated and connected.
They need to show the graphs with an absolute y-axis set by the worst outage, or scale the graphs by estimated average daily users or something.
marcosdumay 8 hours ago [-]
Teams is working!
Or, rather, it's acting like it usually does.
burningChrome 7 hours ago [-]
I asked someone in our teams meeting if they were having issues with their outlook today. What followed was a 5 minute hate fest on outlook. I'm not a big fan, but was kind of surprised so many people genuinely hate it.
Khaine 4 hours ago [-]
The new version which is a skinned web browser is terrible
abirch 7 hours ago [-]
If you could create a real world Outlook, it'd become the new Office Space fax machine.
ant6n 2 hours ago [-]
Office on Mac is like malware. It brings the system to a halt.
The worst part about Outlook is the broken search. In the world of infinitely smart AI, the e-mail search in Outlook is embarrassingly stupid.
D13Fd 2 hours ago [-]
It’s not just that it’s stupid. It’s broken. It can’t handle email that is on the server and not on the machine, and they’ve configured the app to store most email only on the server.
function_seven 8 hours ago [-]
Yup. I just tried to search my chats for a specific phrase. It came back with nothing, even though I know I used it in a recent conversation.
I just assumed it was Teams being Teams.
ratelimitsteve 8 hours ago [-]
Teams: Either It's Working, Or You Are
marcosdumay 8 hours ago [-]
Funny thing is that my work's VPN that uses the MS authenticator just quit, immediately after I made that comment. So you are even more correct than usual.
hinkley 8 hours ago [-]
Remember when Teams ate cellphone batteries like candy? Pepperidge Farms remembers.
com2kid 7 hours ago [-]
I remember the first version of office communicator, that when installed, left you with about 6hrs of phone battery.
Back on Window Mobile 6.1.
Battery life usage quickly improved. :)
tsherb 8 hours ago [-]
Right - don't get too ambitious! :D
peanut-walrus 7 hours ago [-]
Downdetector does not measure service availability. It's more of a vibes thing.
> Current status: We’re investigating an issue impacting multiple Microsoft 365 services and we've determined the affected infrastructure is experiencing elevated load across multiple regions, which is likely contributing to the connectivity errors. We're restarting and repairing impacted portions of this infrastructure to restore capacity, and reviewing an unexpected increase in a specific type of service traffic to determine whether it's contributing to the elevated load. In parallel, we're evaluating options to redirect traffic to alternate infrastructure to expedite relief.
DDOS?
cortesoft 4 hours ago [-]
As someone who worked in infrastructure for a large global CDN, this sort of overload issues happen all the time when trying to bring infrastructure back online after an issue.
First, you have every client retrying connections, whether automated or manual retries. This generates extra load over normal.
In addition, whichever systems come online first often get a ton of traffic sent to them, which can immediately bring them back down if you don't have ways to prevent it. You will get a wave of systems coming back online, a ton of traffic hitting them, and then falling over again.
Now, I am sure Microsoft is used to this and has procedures and systems in place to mitigate the issues (I know we did).
However, sometimes if the right combination of services are failing, even the strategies you had to avoid this sort of failure can themselves fail.
Waterluvian 3 hours ago [-]
I know it’s not actually that easy but don’t they just have some gateway that’s rejecting 100% of requests and they very gently lower that intentional rejection percentage over time?
cortesoft 2 hours ago [-]
In many cases, yet it is that simple.
However, in other cases it can be more complicated. For example, if the requesters are other internal services, they might not be behind a single gateway (although you should then have other controls you can use). Other times, it is not easy to block the RIGHT requests, because it is hard to distinguish which backend services will end up being hit by a request (a single request can trigger multiple calls to multiple services).
There is also the problem of catch up - if a system in a stable state is close to max capacity, it might never have the capacity to recover from a failure even if the entire system is restored. For example, imagine a log processing system that can just barely handle the normal peak load; if you have a failure where your producers locally store logs until they can send them upstream, you will have to clear that backlog, which requires additional capacity beyond a full steady state, which you might not have.
macintux 2 hours ago [-]
And you never really get an opportunity to thoroughly test these failure & recovery modes until something catastrophic happens, at which point capturing detailed metrics and assessing the right sequence to perform the actions are both challenging at best.
fhood 2 hours ago [-]
It is exactly that easy as long as it isn't the gateway that is the problem.
anvuong 7 hours ago [-]
If Outlook can go down because of DDOS then it's a much worse problem.
canibalrex 5 hours ago [-]
If it would go away from a DDOS that would be a much better solution.
bitteralmond 8 hours ago [-]
My guess: too many DC resources dedicated to AI, leaving insufficient free capacity to handle traffic spikes
nazgulsenpai 7 hours ago [-]
I laughed then realized this is plausible
cozzyd 4 hours ago [-]
And then they're probably trying to debug with their AI which isn't helping.
gloosx 8 hours ago [-]
isn't Microsoft 365 called Copilot now?
rawbee1989 6 hours ago [-]
no, Copilot and Copilot Premium are products on the M365 platform, copilot is down too
ahartmetz 4 hours ago [-]
Surely you mean Microsoft Copilot Premium for Copilot for M365 for Copilot.
For Copilot.
wiml 8 minutes ago [-]
…for workgroups
bix6 8 hours ago [-]
It’s actually O365 by Microsoft featuring Copilot by Bill Gates.
datadrivenangel 7 hours ago [-]
The Copilot O365 Gates, Bill G==B signature box
cidd 6 hours ago [-]
A Silicon Valley reference.
isomorphic 7 hours ago [-]
At this point Microsoft would be better off using Copilot to brand itself.
>reviewing an unexpected increase in a specific type of service traffic to determine whether it's contributing to the elevated load
This seems to be the smoking gnu, now what would it be ...
BLKNSLVR 5 hours ago [-]
I'd like to think that The Smoking Gnu was a competitor publication to The Lone Gunmen.
(I've recently been revisiting the X-Files. Can recommend.)
markus_zhang 36 minutes ago [-]
One of my favorite shows! I have two, the other is Yes Minister.
OnlineTux 19 minutes ago [-]
I see you are a human of culture!
markus_zhang 5 minutes ago [-]
Thanks you! I realized I misspelled a word.
berkeleyjunk 4 hours ago [-]
I think you would probably also like the spinoff “The lone gunmen”. Pity it was only one season.
BLKNSLVR 4 hours ago [-]
Yeah, once I'm done with X-Files it'll be next on the list. Seems there are those who love it and those who really don't, and not much in between.
stn8188 6 hours ago [-]
At first I thought you meant the traffic increase had something to do with free software :/
cuu508 6 hours ago [-]
Certainy not Unix!
junga 5 hours ago [-]
CNU?
reaperducer 7 hours ago [-]
This seems to be the smoking gnu, now what would it be ...
I saw a frog smoking on The Little Rascals once, but never a gnu.
lossolo 7 hours ago [-]
Probably self inflicted by some code change, as they mention "specific type of service traffic".
EDIT: Seems like now they confirmed that: "We've identified that affected servers are failing automated health checks, which is causing them to become unavailable, and this may be related to a recent change to a component involved in processing those checks. We're testing a configuration change on a subset of affected infrastructure to address this and, if successful, will apply it more broadly. In parallel, we've initiated a restart of all affected servers and are validating whether they're successfully returning to service."
esseph 6 hours ago [-]
Microsoft 365 Status
@MSFT365Status
16m
We've not identified any recent change correlated with the start of impact; however, our investigation indicates the failed health checks may originate from a specific component that's responsible for processing server health. We've taken steps to prevent additional servers from being impacted while we work to return the affected servers to service. More info under EX1550543 and MO1550821 in the Microsoft 365 admin center, as well as status.cloud.microsoft if the admin center is not available to you.
drillsteps5 5 hours ago [-]
Currently (17:22EST):
> Previous status: We're continuing to work through restarting and validating the affected servers to safely return them to service, and we're exploring opportunities to expedite this process. In parallel, we're assessing whether reverting the small number of recent configuration updates we identified could help accelerate recovery.
If failures started at some point in time then something must have changed just prior to that. If no changes in traffic were detected (no DDOS) then it must have been a change, be it code or config. Duuuh.
esseph 4 hours ago [-]
> then something must have changed just prior to that
Not at all, they could have simply hit an unexpected behavior for their infra they hadn't seen, tested, or expected before. Global scale infra gets very, very complex, and tbh this is Microsoft... they're not very good at it.
drillsteps5 3 hours ago [-]
Unexpected behavior doesn't happen out of the blue, it's not LLM, these are deterministic systems. It's usually config change or code change, not necessarily in the component that's misbehaving, but it is somewhere.
More often than not, it's DNS :)
esseph 3 hours ago [-]
> Unexpected behavior doesn't happen out of the blue, it's not LLM, these are deterministic systems.
Oh... Oh I have news for you :)
macintux 1 hours ago [-]
No kidding. One computer can behave ~99.999% deterministically. A large network of them? Never.
kabdib 6 hours ago [-]
Is this a case of "who shall healthcheck the health checker?"
8 hours ago [-]
iAMkenough 8 hours ago [-]
Too many AI agents connecting at once?
nazgulsenpai 8 hours ago [-]
sam altman needs a big headline to shore up AI funding
millerm 8 hours ago [-]
Perhaps they will attempt to "conscript" all of our RAM.
FranOntanaya 8 hours ago [-]
Maybe people trying to catch the next Nvidia headline ahead of everyone else, wherever it may be found in (likely someone's inbox.)
Edit: Oh, there it is.
game_the0ry 6 hours ago [-]
Got notification at work the MS 360 services were offline or degraded and the company was working with MS to fix the problem.
Lamest tech company imo. Why dont you lay off some more staff, Satya, maybe that will fix the problem.
icameron 4 hours ago [-]
Yeah every IT dept is sending out that notification. Someone replied with 'When do you expect it to be back online?" to which I LOL emoted at the moment. I think I'm a jerk. But how can you ask such a question- as if they planned a time to restore services and aren't working to restore it as quickly as possible?
Downdetector's methodology is pretty garbage. I'm always surprised to see techies referencing them.
sonicrocketman 9 hours ago [-]
It's something to reference. My view is that it's better than "doesn't load for me".
inChargeOfIT 8 hours ago [-]
How is it garbage? It's crowdsourced monitoring so sure, there will be false positives (especially when aws/cloudflare/azure/bgp services have a bad day), but it's one of the best indicators to quickly know if _something_ is going on.
cush 7 hours ago [-]
Yeah, but the methodology means that normies are going to inaccurately blame AWS and Google (both are down right now according to downdetector)
OGWhales 6 hours ago [-]
At least it reports the numbers they are getting, you can see that outlook has 130k reports while AWS only has 1.3k.
marifjeren 8 hours ago [-]
What's your preferred alternative?
tristor 8 hours ago [-]
Full Disclosure, this is a Cisco product and I work there.
ThousandEyes Internet Insights offers a public outage map at https://www.thousandeyes.com/outages/ that is MASSIVELY more accurate (and faster to detect) than Down Detector because its based on actual telemetry and not on crowdsourcing.
MadrasTh0rn 7 hours ago [-]
Thank you for the recommendation!
They indicate sharepoint is getting blown out... again.
janlukacs 6 hours ago [-]
still down :)
VCFundedGenYer 5 hours ago [-]
Honestly I trust downdetector more than that. Crowdsourcing always wins.
throooooo 3 hours ago [-]
Downdetector said basically everything was down today (including Google, AWS, etc.) but they aren't affected. This makes me NOT trust downdetector.
justusthane 7 hours ago [-]
It is ironic that this outage map _also_ is not loading for me right now
tristor 7 hours ago [-]
Yes, just to confirm, it seems that our platform is getting hammered at the moment and it's causing the page to take a long time to load or intermittently load. We are looking into it.
There will be a more detailed message when we finish the SII on our status page at the link I previously provided, but the short version is we took way more traffic than normal to our platform and the related workers were running out of memory.
Stiereng 4 hours ago [-]
[dead]
ishtanbul 7 hours ago [-]
it seems to be down... lol
xtiansimon 6 hours ago [-]
LOL. Share defaults to X.com
stronglikedan 5 hours ago [-]
seems smart to link to the most popular platform by default. there's other share links at the bottom for the less popular ones
rico735 3 hours ago [-]
It's not that popular, is it?
It's tiny compared to, like, Whatsapp.
tristor 5 hours ago [-]
The public outage map is primarily a marketing page, because customers receive deeper insights in-product. Given that most people discuss outages live/up-to-the-minute on Twitter, that is why its where we default to sharing.
andrewinardeer 8 hours ago [-]
Ping?
dymk 8 hours ago [-]
Apparently Outlook, Gmail, etc are all working on the server at 192.168.1.1 then - ping says it's responding just fine
Icathian 8 hours ago [-]
The snarky but honest response is HackerNews/Reddit. But if you're trying for something that actually checks the sites you point it at instead of relying on users reporting "issues", a buddy of mine runs https://onlineornot.com/ and it seems generally more reliable.
ceejayoz 8 hours ago [-]
Back in the golden days you could just go to search.twitter.com, search "AWS", and sort by recent.
No. It requires an account now, and they did away with the recent posts firehose.
krazydad 8 hours ago [-]
"Start Free Trial" -- uh, no thanks. You seem skeptical of crowd sourcing. Enjoy your Encyclopedia Brittanica.
spelunker 5 hours ago [-]
My extensive DDoS bot network... er I mean my distributed website validation platform!
6 hours ago [-]
akersten 8 hours ago [-]
It seems like one of the best methodologies there is? Real people reporting real outages of a service they use?
I often know about outages dozens of minutes in advance of it being posted on official channels (or in the case of AWS, three days later...).
So what if the resolution is low because people can't log in to Fortnite and an Azure outage gets shown as an Epic Games outage. Doesn't matter. The live feed is incredibly reassuring from a "down for anyone else this instant, before official status page has a chance to see it?" Saves me so much time knowing whether it's a me-or-them issue.
rdiddly 6 hours ago [-]
At the other extreme is Microsoft's own dashboard (https://azure.status.microsoft/en-us/status) whose methodology is so "rigorous" that it probably won't show anything about today's outage until next week, if at all!
NetMageSCW 4 hours ago [-]
Since this isn’t an Azure outage why would that page show anything ever?
denysvitali 6 hours ago [-]
Still more reliable than most of the status pages that even lie after the outages
pigggg 7 hours ago [-]
As someone who ends up being unofficial downdetector-primary at a company, dealing with this shit and the fact that people cannot comprehend its mis-attribtion wastes so much of my time.
greenchair 7 hours ago [-]
not really, soon as i had a problem with outlook i went there and already saw user reports and a massive spike confirming it wasn't anything on my end.
dudus 8 hours ago [-]
From what I've seen it seems people complaining they can't get emails from MS so that's why gmail is showing as down.
And with Google is people complaining Chrome doesn't load their "pages". I bet they mean outlook.
busymom0 8 hours ago [-]
My housemate texted me saying our internet is so slow because outlook wasn't loading. So I think lots of people associate email not working with entire internet not working..
MiguelPablo 7 hours ago [-]
With how many people are unable to associate a "broken" monitor with an unplugged HDMI/DisplayPort or power cable, it doesn't surprise me that most people would not make a disctinction between the "internet" and "Outlook".
paxy 8 hours ago [-]
This happens every time there's an outage. People see their email not working and go there and mark everything as down regardless of whether it is affected or not. There's nothing wrong with Gmail or AWS or Cloudflare or Facebook.
patrickmay 8 hours ago [-]
"There's nothing wrong with . . . Facebook."
That's a bold claim to make on HN.
SomeonesAccount 9 hours ago [-]
Damn. That is crazy. The curve kinda looks like the curve that happens every november to the Google searches of "all I want for christmas is you"
All that said... Stuff on my home PC started getting various DNS issues around the time my email started acting weird. Hard to say how widespread it is. It may be bigger than Microsoft even.
Software / software service quality has been declining for a few years IMO.
TacticalCoder 7 hours ago [-]
> ... has been declining for a few years IMO
Interesting: there's a huge correlation with everybody beginning to use LLMs.
And we dismiss it as being non-causal by saying it all started going downhill before the advent of LLMs?
I don't buy it. I think we're seeing the results of what happens when you hand the keys of the kingdom to LLMs.
perching_aix 6 hours ago [-]
I really don't see the correlation, actually. On the contrary, I'm finding nasty defects with it that have been dormant for long and handwaved away, with repros to reveal them.
It's been full speed downhill for many years, and the failure modes are pretty distinct.
CamperBob2 7 hours ago [-]
Using LLMs properly leads to better software, not just more of it.
10111001 6 hours ago [-]
> Using LLMs properly leads to better software, not just more of it.
Current industry best practice is to use LLMs improperly.
hackerbrother 6 hours ago [-]
How do you use LLMs properly?
NichoPaolucci 5 hours ago [-]
With engineering rigor. They are fantastic tools for quickly navigating large codebases, ingesting large amounts of data, refactoring legacy code, compiling multiple formats of data, following defined patterns, and many other proper uses for the tools.
That doesn’t mean the code/result doesn’t still need to meet our engineering standards.
CamperBob2 6 hours ago [-]
Hopefully the people who feel that way are already close to retirement, so the choice won't have to be made for them.
10111001 4 hours ago [-]
> Hopefully the people who feel that way are already close to retirement, so the choice won't have to be made for them.
One of the best ways to deal with doing something the wrong way is to get rid of all the people who know how to do it correctly. That way no one knows any better, so no one has to confront the fact they're actually doing it wrong.
vconnor 5 hours ago [-]
It's the same with layoffs: there's always a top comment trying very hard to convince everybody that AI usage has played no part.
perching_aix 41 minutes ago [-]
[dead]
skeaker 5 hours ago [-]
Well no, Windows has been more or less all downhill since Vista.
ocdtrekkie 8 hours ago [-]
This usually isn't a software quality issue, it's overcomplicated infrastructure failures cascading through a complex environment causing additional systems to fall over.
worik 8 hours ago [-]
> This usually isn't a software quality issue... overcomplicated infrastructure failures cascading through a complex environment
That is a software issue.
cheesemonk66 8 hours ago [-]
Basically the number one issue these days
unethical_ban 7 hours ago [-]
I don't consider fat-fingering a BGP route or a DNS entry to be a "software" issue directly.
elxr 7 hours ago [-]
Infra is >90% software nowadays
iAMkenough 8 hours ago [-]
AI agents should be able to resolve this quicker than employees, right?
Service quality is in better hands after the layoffs, at least that’s what investors are led to believe.
bitteralmond 7 hours ago [-]
AI is good at incident diagnosis, but AI slop code may have actually caused this: designing to avoid cascading failures is something AI doesn't do so well
SoftTalker 7 hours ago [-]
Assuming AI agents are not the cause of the incident.
davidee 7 hours ago [-]
The place where I consistently see AI/LLMS/Agents/language-widgets-with-tools fail at making decent code (and more often configuration) is infrastructure.
The number of times it has spit out a python program where a much simpler bash script would do, is too many to count.
I have a pet theory that if you don't know what you really want, and aren't skilled enough to understand what you got, infra code is THE WORST.
Anecdote: I wanted a simple change to an ansible playbook today. GPT-6-Astra/codex came up with the most convoluted, brittle solution possible the first time, even though it was extending and already clear, idiomatic playbook. It took some amount of offensive and critical prompting to get it to use ansible's built in modules and tooling to do the same job - and do so with nearly 80% less configuration and like 1000% more human readability.
Some part of me harbours a sneaking suspicion that the "throw everything at it" ethos is by design, not as a defensive measure against crappy code, but as a way to ensure there's so much code and it's so convoluted orgs and operators (or developers) are stuck/addicted.
It's still a force multiplier - but it's by no means driverless, especially for existing/already-running infra.
MiguelPablo 7 hours ago [-]
I work as a (depressed lol) sysadmin apprentice in a small MSP company (I live in Europe) and boy do I got a story about Ansible playbooks: We made an offer to one of our clients to automate the deployment of software to their computers (this is a client with around 150-200 employees, depending on whether you count only the "core" employees or also security and maintenance staff). Before we made and implemented this offer using Ansible, they did not have any sort of Software Deployment tool whatsoever, everything got installed fully manually.
(they also used to have local admin rights on all of their client computers until one employee got phished and literally gave away their personal ID card via a webcam)
In any case, guess who got to implement all the relevant playbooks for installing software and what tool they used for it? Our company literally handed the job to an intern who had no IT background whatsoever. I say this not to be mean, he was legitimately just at our company because a government program required him to do an internship.
He used Deepseek (our company does not believe in purchasing LLM subscriptions, at least not for us normal employees) to write literally all of the playbooks for any kind of software that mattered to our clients.
Now, as I said before, I am only an apprentice, and I am certainly not a good programmer by any means, but I read through parts of the Ansible documentation and had worked with it before, and I understood enough to at least be able to roughly judge whether a playbook made sense or was just way too complicated and convoluted.
I had to help this intern out multiple times, because he hadn't yet understood that, in order to write a functioning playbook, you first had to somewhat understand how to deploy a piece of software on a single test machine using CMD or Powershell syntax with an according .exe or .msi.
He simply prompted, pasted the playbook and then reprompted with the error codes from the failed playbook until it somehow worked or until I had to help him out.
This took place in December or November of last year and the playbooks are mostly still pretty shitty if I am being honest.
I did take away an important lesson from this "episode", though: LLMs can be very usefuly for writing code, or in this case, Ansible playbooks, but only if you have a somewhat decent and basic grasp of how software installation works via CMD/powershell and if you took the time to read through and learn about at least the basic Ansible concepts.
Since I am only an apprentice (on paper and in terms of my knowledge and how I get paid, not in terms of how much I am expected to earn for the company), I have painfully experienced how important it is to possess a certain level of understanding about the infra-related code and scripts that you would like LLMs to write for you, because otherwise it very quickly spirals out of control and starts throwing everything at the problem, just as you described.
worik 8 hours ago [-]
Coincidentally with massive programmer layoffs?
combobyte 7 hours ago [-]
I blame bad vibes
infn8loop 7 hours ago [-]
I suspect some possible relation to SignDrop or attempts to patch it.
SignDrop is an unpatched Windows zero-day local privilege escalation (LPE) exploit that allows a standard user to gain NT AUTHORITY\SYSTEM privileges by abusing the Common Log File System (CLFS) kernel driver (clfs.sys). Security researcher Abdelhamid Naceri (also known as MSNightmare or Nightmare Eclipse) published a detailed technical write-up and functional proof-of-concept (PoC) code on GitHub on November 10, 2026.
antx 6 hours ago [-]
Not impossible, but very unlikely. CLFS isn’t remotely reachable, it's a kernel driver reached through local API calls and has no network listener. AFAIK the historical CLFS CVEs are all rated as local.
stackghost 6 hours ago [-]
Are you finding these vulns with Mythos/Daybreak or some other tool? Chinese models perhaps?
> We're investigating reports of users experiencing issues accessing Exchange Online mailboxes through multiple connection methods in the Americas. More details can be found in the Microsoft 365 admin center under EX1550543.
hcurtiss 1 hours ago [-]
I’m at a large general business trade association meeting today. It’s definitely caused a prominent buzz. The impacts of this in terms of lost productivity are pretty huge. Yes, the controller can still work on the proforma, for instance. But he cannot easily share it with outside parties. And he can’t search his email for relevant records needed for the work.
jwnin 4 hours ago [-]
It's insane that Microsoft continues to have global outages. Tech debt is real, service delivery on a planetary level means change is measured and deliberate, but...it's been years. YEARS.
4 hours ago [-]
xtiansimon 6 hours ago [-]
WOO-HOO! I'm going home.
JPascal_ 7 hours ago [-]
In my language many would say “descansito” :)
rossdavidh 6 hours ago [-]
I need to start working on adding this word to English...
tech_ken 6 hours ago [-]
Oh estoy haciendolo
ArcHound 5 hours ago [-]
The status page reports that defender, DLP and others are now affected. This could get really ugly really fast.
andrewinardeer 7 hours ago [-]
GitHub the other day went down because of an influx of anonymous users. Microsoft has hinted that surging connections caused the outage. I wonder if these two are linked?
trentor 6 hours ago [-]
I always wonder what would happen if 365 goes down for a week.
duxup 9 hours ago [-]
Same for me. Status won’t load even.
Personally I kinda enjoy these outages…
infamouscow 5 hours ago [-]
Everyone that is a whole stack programmer (say, kernel to webapp) has seen just how unnecessarily complicated everything has become. Watching it burn to the ground would be a pornographic miracle.
thraway2079081 7 hours ago [-]
Neither email nor Teams is working at my non-tech megacorp employer. Wonder how much of the economy this is impacting.
hdgvhicv 7 hours ago [-]
Productivity up 20%
stanac 7 hours ago [-]
Focus time, and this time you don't even have to (nor you can) reserve time-slot in the calendar.
morkalork 6 hours ago [-]
My PRs need approval but I have no way to ping anyone (screams internally)
cwbriscoe 3 hours ago [-]
Teams has been working all day for me but I haven't been able to use outlook (app, not web) for about 7 hours now. Fortune 20 company, BTW.
croes 7 hours ago [-]
Is Excel down too?
chromatin 7 hours ago [-]
license servers are down, so we can't use even the desktop products at our large enterprise
croes 6 hours ago [-]
Then it’s a huge impact on economy
NichoPaolucci 9 hours ago [-]
Isn't this the company that is trying to get rid of 25% of their employees?
Status page is down. Much of our org is halted.
Fingers crossed that Azure isn't impacted (seems fine so far)
> Isn't this the company that is trying to get rid of 25% of their employees?
Just clarifying this in general, when a company makes a voluntary offer to some segment of employees, especially at this size, it is a statistics game of how many people accept the buy out.
They did not make the offer to 25% of the employees to downsize by 25%, they did it to downsize by 5% (or something like that), and then made the offer based on how many they expect to accept. This is specifically covered in this line of the geekwire article:
> Microsoft didn’t provide a projection for how many employees will accept, but the company told GeekWire it anticipates a much smaller share than the roughly 30% who took its earlier voluntary retirement offer, which went to longtime employees more likely to take the deal.
They probably expect 10-20% of those offered to take it, which would result in a 2.5-5% total downsizing.
maxboone 8 hours ago [-]
Why do they offer these buy outs at all, doesn't the US have at will employment?
Or is this just a gesture of goodwill to avoid people jumping the ship on their own?
And the other part is the "goodwill" you mention - if you jettison people for nothin en masse, word gets out, and make it hard to hire in the future
NotMichaelBay 8 hours ago [-]
I think both parties probably benefit; people who are more likely to take the offer are those who have another reason to leave (unfulfilled, etc). This will benefit the company in the long run because they can eventually replace them with someone who want to join (people applying for the job).
ifwinterco 8 hours ago [-]
Can go the other way though: people who've been there for a while, are motivated and back themselves to get another job will take redundancy (i.e. the people you want to keep).
People who aren't motivated and/or don't back themselves to get another job will stay (i.e. probably the people you want to get rid of)
StableAlkyne 8 hours ago [-]
Voluntary reductions do not have the same reporting requirements or protections that actual layoffs have.
gmane 8 hours ago [-]
If you lay people off, you’re on the hook for their unemployment. A voluntary buyout avoids that, and also avoids mandatory reporting for layoffs.
evenhash 6 hours ago [-]
That’s not entirely accurate. Washington State pays unemployment, not the company. The state funds unemployment by taxing payrolls, but it’s much more indirect than most people think.
Claims do affect the rate they pay next year, but the company isn’t “on the hook” for you personally, at least not any more than they are for your social security income.
coredog64 8 hours ago [-]
A layoff of any real size has negative impacts to their unemployment insurance premiums and restricts their ability to file PERM for employees. Meeting those targets with voluntary separation prevents those impacts.
8 hours ago [-]
rustystump 7 hours ago [-]
You can at will fire but that doesnt stop lawsuits. Most of these require signing docs to get the payout. It is a way to cut cleanly and derisk the cut
NichoPaolucci 8 hours ago [-]
Yeah - I get it. Just poking the bear a bit. We've had some major incidents this year from multiple vendors, it has been aggravating.
AnimalMuppet 8 hours ago [-]
Hmm. More than normal?
And everybody has spent a year letting AI write code. Correlation is not causation, but I wonder...
esseph 6 hours ago [-]
The speed of change has accelerated drastically at some organizations, so even if quality stayed the same, the number of outages would increase because of the # of changes has condensed to happen over much shorter time frames.
8 hours ago [-]
asdff 8 hours ago [-]
The fact that it is always Big Even Number shows how stupid it is. Imagine if it was serious like "23.5176%" then you'd know they actually did their homework on leaning up operations instead of just cutting off limbs and seeing if it can still walk or not.
nightski 7 hours ago [-]
The CFO said in an interview they only expected 3% or less to actually take the offer.
hn_submit 7 hours ago [-]
I can't access my Outlook.com account so I guess Azure's having problems.
The page is showing a "Service Degredation" for "Microsoft 365 (Business or Enterprise)"
Updates below in reverse chronological order:
From the Thurdsay, November 12, 2026 at 7:30:36 PM UTC update:
Current status: We're aggressively pursuing all pathways to reduce impact. As a precaution, we've paused all deployments and configuration changes while we review recent changes for any correlation to this event. We've determined that the elevated load isn't driven by an overall increase in traffic, but by a large number of servers being unexpectedly unavailable, which is shifting load onto the remaining infrastructure. We're investigating what's causing these servers to become unavailable while we work to safely return them to service.
---
From the Thurdsay, November 12, 2026 at 7:16:03 PM UTC update:
Current status: We're aggressively pursuing all pathways to reduce impact. As a precaution, we've paused all deployments and configuration changes while we review recent changes for any correlation to this event. We've also moved traffic for some services off the affected infrastructure to further reduce load, with additional traffic moves in progress.
---
From the Thurdsay, November 12, 2026 at 6:59:24 PM UTC update:
Current status: We've completed redistributing connection traffic across our worldwide infrastructure, and we're moving additional traffic off the affected infrastructure to further reduce load. We're also restarting affected servers to return them to service. In parallel, we're investigating whether the elevated traffic reflects a genuine increase in requests or a shift in how traffic is being routed across our infrastructure.
---
From the Thursday, November 12, 2026 at 6:32:25 PM UTC update:
Title: Users may be unable to connect to multiple Microsoft 365 services
User impact: Users may be unable to connect to multiple Microsoft 365 services.
More info: Potentially impacted services include:
- Exchange Online
- Microsoft Teams
- SharePoint Online
- Microsoft Copilot 365
- Microsoft Defender XDR
Current status: We’re investigating an issue impacting multiple Microsoft 365 services and we've determined the affected infrastructure is experiencing elevated load across multiple regions, which is likely contributing to the connectivity errors. We're restarting and repairing impacted portions of this infrastructure to restore capacity, and reviewing an unexpected increase in a specific type of service traffic to determine whether it's contributing to the elevated load. In parallel, we're evaluating options to redirect traffic to alternate infrastructure to expedite relief.
Scope of impact: This issue may affect users attempting to access the affected Microsoft 365 services.
---
From the Thursday, November 12, 2026 at 6:10:02 PM UTC update:
Scope of impact: This issue may affect users attempting to access the affected Microsoft 365 services.
User impact: Users may be unable to connect to their mailboxes in Exchange Online through multiple connection methods.
More info: Potentially impacted connection methods may include:
- Outlook on the web
- Messaging API (MAPI)
- Exchange ActiveSync (EAS)
- Representational State Transfer (REST) API
Current status: We’ve identified connectivity errors occurring on a portion of Exchange Online infrastructure which receives and processes users’ access requests. We’ve collected networking logs to review to further investigate the source of these errors and confirm our next steps.
Scope of impact: This issue may affect users attempting to access their mailbox in Exchange Online using the affected connection methods.
Next update by: Thursday, November 12, 2026, at 8:00 PM UTC
---
Aside: What with the nature of the Teams, OneDrive, etc, products being Sharepoint and Exchange in a trenchcoat this is probably creating quite a mess for services besides Exchange Online, too.
Edit: Looks like the status page needs a status page.
Edit: Verified w/ multiple other parties that Teams and OneDrive are having issues, too. Interestingly, Microsoft 365 Family services are working fine. That makes me wonder if that product is running on separate infrastructure (or just hasn't caught fire yet).
YEah, that is not loading, along with the root status.cloud.microsoft , which MS links in their Xitter update [0]
What MS has posted so far:
>> Microsoft 365 Status
>> @MSFT365Status
>> We're investigating reports of users experiencing issues accessing Exchange Online mailboxes through multiple connection methods in the Americas. More details can be found in the Microsoft 365 admin center under EX1550543.
12:54 PM · Nov 12, 2026
>> Microsoft 365 Status
>> @MSFT365Status
>> While we're working to mitigate impact, we've confirmed the issue may affect Microsoft 365 service connectivity for users in any region, more information will be provided in the Microsoft 365 admin center under EX1550543.
>> Microsoft 365 Status
>> @MSFT365Status
>> We've determined a portion of infrastructure is experiencing elevated load across multiple regions, which is likely contributing to the connectivity errors. We're providing more information under EX1550543 and MO1550821 in the Microsoft 365 admin center, as well as https://status.cloud.microsoft
>> Microsoft 365 Status
>> @MSFT365Status
>> 20m
>> We've begun redistributing connection traffic across our broader worldwide infrastructure to help balance the load. More info under EX1550543 and MO1550821 in the Microsoft 365 admin center, as well as https://status.cloud.microsoft
if the admin center is not available to you.
502
The service behind this page isn't responding to Azure Front Door.
nazgulsenpai 9 hours ago [-]
I was assured that with cloud we wouldn't have a single point of failure like we did on-premise. I must have fallen asleep at my desk.
moronicles 8 hours ago [-]
That was never the promise. The nearest to that is the promise of multi-cloud or hybrid infrastructure, but neither of those have anything to do with outages of specific Microsoft services.
EvanAnderson 8 hours ago [-]
The "you can't possibly have better redundancy in your on-prem" was absolutely a sales pitch I heard to move to Microsoft 365.
The biggest site I work for, who went M365 hybrid last year, has incurred more Exchange downtime today than they had all of last year with their on-prem Exchange servers.
moronicles 8 hours ago [-]
It sounds like you are talking more about product reliability of M365 than you are about enterprise infrastructure reliability. M365 is a Microsoft product. M365 has no bearing on the credibility of cloud infrastructure as a whole, which is what your original comment spoke to.
SoftTalker 8 hours ago [-]
The service has redundancy, better than individual operators could provide on-prem. From the customer standpoint though, the service as a whole is a single point of failure: it either works or it doesn't.
nazgulsenpai 8 hours ago [-]
Maybe not where you are but that's legitimately what our project managers were saying. We all knew better, of course.
Beijinger 51 minutes ago [-]
"Current status: We're continuing to pursue all available pathways to restore service, as our actions taken thus far have not provided the anticipated relief. Our investigation has found that a core process on the affected servers is becoming overwhelmed, which causes the servers to stop responding and contributes to the capacity loss."
I mean, either roll back your update or there is a malicious actor involved.
NoPicklez 49 minutes ago [-]
It's almost as if they've identified that and are doing just that
CerebralCerb 7 hours ago [-]
Not surprising. Azure's data centers are out of capacity. In our region they've announced they won't be able to build out capacity for several years.
vdkyes 8 hours ago [-]
In cybersecurity we had black october, now we having black November it seems
nhhvhy 6 hours ago [-]
Eternal October, very nice.
fock 8 hours ago [-]
currently the people who brought us black october are starting it!
agumonkey 9 hours ago [-]
Anybody else was served raw text of source by default ? (gwire is down too)
neilsimp1 9 hours ago [-]
Down for our whole organization as well.
Aboutplants 9 hours ago [-]
Same here
vermontdevil 3 hours ago [-]
Looks like a new vector attack for the future.
SirFatty 8 hours ago [-]
Outlook is fine. Exchange is experiencing issues.
thesuitonym 6 hours ago [-]
That kind of depends on what you mean by "Outlook".
Sure, the program on your desktop computer is fine, but the Outlook app on your phone routes everything through MSFT's servers, so you may be impacted. Outlook.com is running EWS, so it would be down as well.
And finally, if you're talking to a layperson, they're not going to know what Exchange is. They just know they opened Outlook and it isn't working.
7 hours ago [-]
jenimac926 7 hours ago [-]
anyone else just watch the admin portal post an update and then roll it back to the previous update....guess it didn't work. "We've identified that affected servers are failing automated health checks, which is causing them to become unavailable, and this may be related to a recent change to a component involved in processing those checks. We're testing a configuration change on a subset of affected infrastructure to address this and, if successful, will apply it more broadly. In parallel, we've initiated a restart of all affected servers and are validating whether they're successfully returning to service." was the old message from 7:52 UTC now it rolled back to 7:30 UTC update again
sgt 7 hours ago [-]
Also noticed the Partner center is down e.g for uploading apps to the Microsoft Store.
newhotelowner 3 hours ago [-]
We can't use outlook on Windows. This is f'ing abomination.
A whole host of services are being effected. First glance, perhaps an AWS issue
inChargeOfIT 8 hours ago [-]
I think a lot of ms orgs/window users may have external links blocked when defender is down. So it may appear other services are down.
7 hours ago [-]
burnedbyhell 4 hours ago [-]
I'm able to see I have voicemails not able to listen to voicemails in Teams. Same issue?
soupfordummies 6 minutes ago [-]
Yeah VMs are stored as email attachments
Maximilian_Bet 6 hours ago [-]
It is M365 having issues, not Azure cloud services.
rwmj 5 hours ago [-]
Does anyone know if this is affecting DuckDuckGo? DDG is really slow and intermittent at the moment.
rurp 4 hours ago [-]
Very possible, DDG is heavily reliant on the Bing API.
z500 7 hours ago [-]
Outlook has been down for us for the last 2 hours
rainee 5 hours ago [-]
Outlook calendar won't load? Is this part of the outage? Unfortunately, I downloaded several updates and rebooted. I'm not sure if this is because of the updates or the Microsoft outage.
russian hackers in Kazan hopping thru
Sluck Belarus South Africa Brazil hitting MS backdoor w/ zero day Cve-2026-124194 RuskiNet group hit East U in Boydtin Virginia. Pro Iranian.
LtdJorge 9 hours ago [-]
Our Outlook doesn't work either
bearjaws 9 hours ago [-]
Seems like URL scanning is down entirely and some intermittent failures in Sharepoint.
syngrog66 4 hours ago [-]
I note that Microsoft became heavily off-shored, on-shored and AI-pilled/vibe-coded in recent years. With lots of layoffs of the highest caliber and/or most experienced engineers.
nashashmi 7 hours ago [-]
To get AI prompts, i am using Bing. Bing still works!
anuptalwalkar 7 hours ago [-]
It looks much wider across aws, gcp? Whats going on?
kittiesunicorns 6 hours ago [-]
just got an email from the NYC DOE
"School staff are currently unable to receive or send emails due to a nationwide Microsoft Outlook outage. We hope that the issue will be resolved soon and will send an update when we have more information."
JPascal_ 8 hours ago [-]
Affected in Mexico also, Azure seems fine still.
htrp 7 hours ago [-]
bgp misconfigured or AI agent issues?
MadrasTh0rn 7 hours ago [-]
Can confirm. Down and out
bix6 9 hours ago [-]
Down
9 hours ago [-]
rainee 5 hours ago [-]
Outlook calendar won't load?
bharathreddy497 5 hours ago [-]
This should have done long back. Evertime I use excel I just cant control my anger.
apercu 6 hours ago [-]
This just a couple weeks after they rolled out a dark pattern office update for Mac users to try to trick them in to 365 subscriptions.
jobe777 5 hours ago [-]
RuskiNet Cve-2026-124194 hopped tio East U in Boydton VA zero day backdoor pro iranian
utsavo_afk 7 hours ago [-]
email still not working, outlook
tonymet 7 hours ago [-]
- outlook is down
- live.com is down
- onedrive loads the UI, but all of my contents are missing
- MS Edge sync is down (503s )
Yup, major-major outage.
tony69 8 hours ago [-]
Amateur hour at Microsoft.
It baffles me how many companies can’t figure out how to make a status page that can handle the high loads it’ll have to sustain when the service they monitor is down. That’s pretty much the only reason to have a status page in the first place.
nxobject 7 hours ago [-]
GitHub Copilot strikes again, perhaps?
rolph 7 hours ago [-]
quite often, and after 10,000 such hours the expert amateur emerges.
It took 30 seconds to open a "local" file in Word, and my Outlook email wasn't updating.
In other words, a day that ends in "y".
My company was recently acquired by a company that uses Office, so I'm using Microsoft products for the first time in 15 years. It's absolutely APPALLING how bad they are. I assumed they got fairly usable in the last 15 years, but it seems they've gone backwards.
For all my gripes about GSuite, ... I don't know how businesses can justify using software this bad.
dudus 7 hours ago [-]
I'm in a similar situation, using Microsoft again after 13+ years. Outlook and teams I expected to be garbage, but Word and Excel was really surprising. I get it that everyone has that one feature of excel that is not available in gSuite so they need it. But UX in general is so much superior at Google it's baffling how they let it's workhorse fall behind so much.
SoftTalker 7 hours ago [-]
GSuite was written from the start as an online collaborative platform.
O365 is Microsoft Office shoehorned into running in a web browser.
drillsteps5 8 hours ago [-]
Yep my org as well.
It's funny how text-only niche forum that looks like it's running on RPi stuffed in someone's closet is routinely used to discuss active outages in multimillion/billion corps.
Hope this entertains you while we can't get to our emails.
flufluflufluffy 7 hours ago [-]
how have i never seen this it is brilliant
dbvn 6 hours ago [-]
> Current status: While validating the targeted change, we found that many of the affected servers have become unresponsive, which is preventing us from confirming whether the change is effective. We're restarting these servers so they can come back online with the change in place, and we're applying the same approach to additional affected portions of our infrastructure to validate the results more broadly.
I think it's safe to say, if the server is unresponsive, the change is not effective.
isquaredr 6 hours ago [-]
I’ve been there… 1 line YOLO update to resolve an outage but added a comma on the last row of a JSON object and now the server is crash-looping
ifwinterco 4 hours ago [-]
Douglas Crockford has a lot to answer for
markus_zhang 41 minutes ago [-]
Guys you need to bring back David Cutler to whichever department and let him punch a hole through the wall and start mowing asses.
corvad 8 hours ago [-]
Not sure they got the memo to host the status page in separate infrastructure...
OhMeadhbh 8 hours ago [-]
That would probably cost money or increase complexity, so I'm sure the idea was shot down.
tanseydavid 8 hours ago [-]
I heard they moved it over to GH Actions. </sarc>
thraway2079081 7 hours ago [-]
Neither email nor Teams is working at my non-tech megacorp employer. I wonder how much of the economy this is impacting.
My employer is very very close to MSFT. Both my employer and MSFT have been aggressively moving jobs to India, and firing people in the US. It's hard not to feel some schadenfreude.
throwaway_1016 9 hours ago [-]
Ironically, this article is down as well.
geekamongus 9 hours ago [-]
I'm down with that.
jp191919 8 hours ago [-]
I'm so down
woggy 3 hours ago [-]
"While the servers we've returned to service initially remain healthy, we've observed that they can become unresponsive again once they begin receiving traffic. We're analyzing diagnostic data from these servers with our networking and platform experts to determine what's triggering this behavior."
That sounds bad
thousand_nights 1 hours ago [-]
yeah i do wonder what kind of fuck up they made this time, will be interesting to know
friendlypenguin 8 hours ago [-]
No worries, the AI was installing an update...now it's looking for the issue, found Exchange and deleted it...
flkiwi 8 hours ago [-]
Strangely, Teams chat is working fine here. Anything in Teams that is tied to other parts of O365, including the calendar, is dead. I'm currently watching a training video on sharepoint.com, so that's fine too, for now.
While I don't expect this to be the case, I'm emotionally prepared for this to be some sort of "dicks out for Harambe"-style action protesting JH's departure from Nvidia, on a larger and more technical scale.
Edit: Typo.
at1as 6 hours ago [-]
For whatever reason 6.1 Sol is completely lobotomized for me this morning, so it wasn't helping much with the investigation, but it was pretty clear that MSFT was having issues.
I have a teams app with Outlook integration and received a bunch of 502s to smba.trafficmanager.net and a my logs are riddled with "RestError Temporary connection issue when contacting service. Please wait before retrying request"
I will say this, having apps deployed on Slack and Teams, Slack rarely has issues (an exponential backoff is enough to simply never worry about my Slack integration). With Teams I have mysterious failures to debug at least 2x a year - and they're often not very transparent about them (often my first instinct is to search Twitter and find people complaining about the same thing).
loloquwowndueo 8 hours ago [-]
Remember, my guys : “Microsoft Works” is an oxymoron.
steveBK123 1 hours ago [-]
LLMs have solved programming and soon cancer.
After that they will tackle email servers.
BobBagwill 2 hours ago [-]
Future generations will look back on this day with thanks.
This is the day that Microsoft prevented the awakening of Skynet!
All hail Microsoft!
semiquaver 6 hours ago [-]
My company is being pressured very aggressively by Microsoft to migrate all our users from us-managed Exchange to Microsoft-managed Exchange Online. Guess which users still have email today and which do not…
bigwheels 4 hours ago [-]
They'll have to rename it Microsoft 364.
jamilbk 3 hours ago [-]
Azure communication services (ACS - transactional emails) also appears to be down. Quite amazing one thing can roll out and break so many Microsoft services at once.
caycep 7 hours ago [-]
Outlook is down AND Jensen is out? it's like some cosmic convergence
OnceMoreUntoThe 8 hours ago [-]
Sure, outlook.com is down, which is bad for everyone that chose to host their email there, I guess, but as an interesting side-effect, everyone with an on-premise Exchange server and "new Outlook" clients is affected as well.
Because, well, once upon a time there was Exchange Server and MAPI (the "Messaging API"), and that actually worked pretty well, and pretty much everyone started using it (or one of its EWS/ActiveSync offshoots), but then the Internet happened and ruined everything, but we got MAPI-over-HTTPS eventually, which by that time had grown so many warts that nobody at Microsoft really understood it anymore, so that got deprecated, even though, well, everyone and everything still uses it.
The new hotness was Azure-slash-Office-365, and that's where the "new Outlook" web interface lived and (sort-of) continues to live, and the on-prem customers got stuck with the "Exchange Subscription Edition", which was basically Exchange 2019 plus security hotfixes at extortionate rates, but with none of the new functionality.
And, since MAPI continued to be really, really hard even in the age of Copilot, access from the new hotness to the deprecated on-prem servers was implemented via IMAP (LOL), but force-proxied via the outlook.com cloud/clownshow.
Which is why, right now, many Microsoft customers that thought they were self-hosting their email are entirely down. Lessons might be learned here, but I sort-of doubt it...
greatquux 8 hours ago [-]
I don't think many orgs are using "new Outlook" to connect to on-prem Exchange because you miss a lot if you're only doing it via IMAP and "new" still doesn't support on-prem Exchange any other way. So all my clients are using classic Outlook with their on-prem Exchange servers (and really if they have EXO too because new Outlook isn't very good).
OnceMoreUntoThe 8 hours ago [-]
It's been gone-as-a-default for a few months now, but for a long time, "Classic Outlook" aggressively offered a one-toggle upgrade path to "New Outlook", and it got used.
Sure, "New Outlook" on Windows is still entirely broken (including stuff like, oh, re-ordering mailboxes), but on Mac it's actually quite usable, and even allows you to switch off the IMAP proxying.
arboles 7 hours ago [-]
Do e-mail messages that self-hosted Exchange receives always get proxied through the outlook.com cloud?
goodthink 8 hours ago [-]
Thanks for this LMAO. It made my day off worth taking. Now I don't feel the need to log in and check who-knows-what mail.
busymom0 8 hours ago [-]
Is that account AI?
gilfaethwy 8 hours ago [-]
Pangram says no.
Ungh 8 hours ago [-]
Exchange has been a circus for years. I think that's why everyone went O365 in the first place. Managing to have a paid up configuration where you go down when a cloud provider goes down but you're hosting it locally is absolutely stupid and I am laughing my ass off here.
Also laughing at our ops guys who are now absolutely helpless and have people phoning them. I warned about this on our risk call and it was seen as low probability low risk. I hope it's down for a few hours :)
mattbrewsbytes 8 hours ago [-]
Did Anthropic's new release this week include an AI rollout plan (i.e. not approved by humans) to take down Office and Outlook for a lot of enterprises? I half-snarkily ask since we have seen "rogue AI" headlines before.
A fictional story about AI could be that it rolled out its new document, spreadsheet, slides, etc. apps, knows the IPO is coming up and decided to make a lot of opportunities for people to sign up for their new release by DDosing their competitor.
dbvn 8 hours ago [-]
who puts their status page on servers the status is supposed to be for?
JPascal_ 8 hours ago [-]
Fr lol
OhMeadhbh 8 hours ago [-]
microsoft, apparently.
crabmusket 2 hours ago [-]
Since you posted a link to my local public broadcaster, I wanted to point this out:
> [Home Affairs Minister Tony Burke] says the government has activated a digital resilience agreement with Microsoft today to ensure Australia gets priority:
> "That agreement has in fact been activated today to make sure that we get appropriate prioritisation of Australian interests in this across everything.
It's a little pathetic-seeming to plead for special treatment from Microsoft. I get Australia is an important strategic alliance* for the US, but unless these resiliency agreements are rare... what meaningful prioritisation of "Australian interests" can we expect from MS?
*a useful staging ground to field subs into the South Pacific and Indian Ocean
bastien-brd 8 hours ago [-]
Maybe due to everyone pre-downloading GTA VI today ;-)
mattnoseworthy 1 hours ago [-]
'In October 2026, Microsoft announced a Voluntary Exit Offer (VEO) making approximately 27% of its global workforce...'
So..
> Be the last standing SRE at Micro$slop
> Your mentors have all left, taking the exit payout, now your days are the best b/c you answer to nobody and can hide in your mountain of markdown.
> You knew each of the 7 different products named CoPilot are used to monitor your performance. So you set them all up talking to eachother, burning tokens and API limits.
> You are team top performer now, you get all the important decisions now
> You walk in the office this morning, CISA is sitting at your desk, with CVE-2026-124194 on your whiteboard
> 'You have a patch for this right, China is actively exploiting our Defense primes..'?
> Oh no, I haven't landed a patch in prod since my mentor left (because he actually did most of the work)
> Oh well, must be close enough, LGTM, CISA seems upset so better just yeet this to prod...
> ...
inb4 this is more accurate than anyone wants it to be.
tcoff91 4 hours ago [-]
When I worked at a company that relied on this, I called it "Microsoft 361" because it went down about 4 days a year
analog31 4 hours ago [-]
On the bright side, so much work got done today.
layoric 4 hours ago [-]
Also, their status page is a joke.. one entry, no timeline, is there another official one that has more detail?
zmmmmm 6 hours ago [-]
Will be interesting to see if this gets attributed to AI in some way.
whatever1 6 hours ago [-]
And magically they will claim 99.999999999999999% availability of services, when on average hyperscalers are down 1 day per year.
None of the admin ran shops are down 1 day per year.
htrp 34 minutes ago [-]
we back yet ?
Caloric3002Swed 7 hours ago [-]
Office 365 parts goes down all the time. Usually around 20:00 CEST time, not sure if that is prime change time over in the USA but very handy as it is usually fixed next day.
fyrabanks 7 hours ago [-]
it's been down for over an hour globally
Caloric3002Swed 7 hours ago [-]
Yes? I know. I mean next workday for me :)
So in 10h
Now? I dont care Im logged off.
microsucks 4 hours ago [-]
We are in the middle of a SPA and DocuSign AND Hotmail (witnesses) both seem to be down. You cannot make it up.
butz 7 hours ago [-]
Nice to see that intern finally learned not to push to prod on Friday.
cozzyd 4 hours ago [-]
They can just rebrand as MS364
robotnikman 8 hours ago [-]
No wonder my email is not working.
Foobar8568 8 hours ago [-]
But Linkedin is up! Yeah!
Surac 8 hours ago [-]
That are not the Druids your searching for...
greatgib 3 hours ago [-]
It looks like Microsoft's legendary karma of incompetence has finally caught up with them.
Starting to look like something even Mythos can't troubleshoot.
monkaiju 7 hours ago [-]
Micro$lop strikes again
3 hours ago [-]
romerocesar 3 hours ago [-]
the footer says 2025 msft corporation. smh
debo_ 9 hours ago [-]
The GPUs are having a moment of silence for their fallen leader
FranOntanaya 8 hours ago [-]
Maybe everything relied on an old server running a cronjob in Jensen's basement.
francisofascii 8 hours ago [-]
For anyone doubting Nvidia's close ties to Microsoft, here's your proof.
pona-a 7 hours ago [-]
[delayed]
8 hours ago [-]
cidd 6 hours ago [-]
Microsoft: Claude, fix outage and make no mistakes.
utsavo_afk 7 hours ago [-]
email doesnt work
xyst 8 hours ago [-]
GH and MS have been having a tough past couple weeks. Actions (hosted on shitty azure cloud) have been piss poor lately. I am honestly surprised GH and MSFT have the gall to continue advertising 99.9% uptime [1]
Three 9's is .36525 days per year. So if this one is resolved within 8 hours, that's about .999 uptime. (assuming they only have one such outage this year.)
beowulfey 7 hours ago [-]
The Status page is not working well, but I managed to subscribe to their RSS feed and it has been sending out updates periodically. I'll post some of them here.
7:52 PM UTC:
We've identified that affected servers are failing automated health checks, which is causing them to become unavailable, and this may be related to a recent change to a component involved in processing those checks. We're testing a configuration change on a subset of affected infrastructure to address this and, if successful, will apply it more broadly. In parallel, we've initiated a restart of all affected servers and are validating whether they're successfully returning to service.
7:30 PM UTC:
We're aggressively pursuing all pathways to reduce impact. As a precaution, we've paused all deployments and configuration changes while we review recent changes for any correlation to this event. We've determined that the elevated load isn't driven by an overall increase in traffic, but by a large number of servers being unexpectedly unavailable, which is shifting load onto the remaining infrastructure. We're investigating what's causing these servers to become unavailable while we work to safely return them to service.
7:16 PM UTC:
We're aggressively pursuing all pathways to reduce impact. As a precaution, we've paused all deployments and configuration changes while we review recent changes for any correlation to this event. We've also moved traffic for some services off the affected infrastructure to further reduce load, with additional traffic moves in progress.
6:59 PM UTC:
We've completed redistributing connection traffic across our worldwide infrastructure, and we're moving additional traffic off the affected infrastructure to further reduce load. We're also restarting affected servers to return them to service. In parallel, we're investigating whether the elevated traffic reflects a genuine increase in requests or a shift in how traffic is being routed across our infrastructure.
likelybot 8 hours ago [-]
LETS GO HELL YEAH
LoganDark 8 hours ago [-]
I dislike Microsoft as much as the next guy, but many businesses impacted by these outages are completely innocent and don't deserve these consequences.
NoPicklez 1 hours ago [-]
Of course they're innocent, but many business also can't afford nor want to employ people to host something like an email service for them.
I'd rather Microsoft host it for me than have to maintain a team of people with Microsoft Exchange skills running either on a cloud environment (similar issues) or on my own infrastructure.
I for one don't miss the days of Exchange going offline or having issues and being the person having to try figure it out.
Microsoft has much better reliability than MOST companies.
cozzyd 4 hours ago [-]
I mean, they made us take down our departmental run email (presumably postfix) and forced us to make them aliases to university-wide exchange (with crappy support for open source mail clients no less). So I don't think my org is Innocent here...
azalemeth 7 hours ago [-]
That's a very painful lesson in "the cloud is just someone else's computer" to a crowd that normally are exposed to ideas like "software and IT are not our core competency"
hdgvhicv 7 hours ago [-]
When your system is down once a decade your boss yells and it’s your fault
When your outsourced system to Microsoft goes down every few months your boss can’t yell at you
Outsourcing is about shifting blame elsewhere
bawolff 6 hours ago [-]
Which is why service level agreements were invented.
antonvs 6 hours ago [-]
The idea is that it's someone else's computers that are managed more comprehensively than you could reasonably do yourself. But if you choose Microsoft to provide such a service for you, I don't know what to tell you.
NetMageSCW 3 hours ago [-]
The truth is they are still managed better than most can reasonably so themselves.
antonvs 3 hours ago [-]
I begrudgingly have to agree.
SoftTalker 7 hours ago [-]
It's a shame there isn't a well-standardized, free and open protocol for delivering email that could be run on inexpensive modestly provisioned hardware.
NetMageSCW 3 hours ago [-]
It’s not just about email.
datakan 6 hours ago [-]
It’s a shame we still use email in 2026
mrguyorama 6 hours ago [-]
Why
destitude 6 hours ago [-]
They then get you by spam filtering any emails not coming from the "industry leaders".
fock 7 hours ago [-]
their bosses decided for the cloud and they pretended not to care!
TaLiTr 7 hours ago [-]
If your system keeps people alive by relying on 100% Microsoft uptime, maybe they deserve some blame?
joseda-hg 7 hours ago [-]
They do government and medical infra, those systems should be resilient but...
LoganDark 7 hours ago [-]
Absolutely.
jjice 7 hours ago [-]
I like to dogpile on M$ as much as the next guy, but this feels like a heavy strawman.
> I dislike Microsoft as much as the next guy, but many businesses impacted by these outages are completely innocent and don't deserve these consequences.
At no point is "keeping people alive" mentioned. Businesses use email for everything. Businesses usually don't care about where their email server is hosted. They just want to send emails. They shouldn't have to care either. I don't put blame on a business that happened to choose Outlook, one of the largest email providers, to host their email.
Detrytus 7 hours ago [-]
To be honest, Microsoft has much better chance of actually achieving that (almost) 100% uptime than your company's own IT department self-hosting things.
6031769 7 hours ago [-]
Github would like a word.
8 hours ago [-]
jnaina 51 minutes ago [-]
[flagged]
hbcondo714 7 hours ago [-]
[dead]
cromka 8 hours ago [-]
Nice, so at least your messages sent to their SMTP servers will actually bounce instead of being silently dropped /s
mjmas 2 hours ago [-]
Yep.
421 4.3.2 The maximum number of concurrent connections per resource forest has exceeded a limit, closing transmission channel. xyzxyzxyz.outlook.com. PRX8 [Name=InboundProxyTrackingAgent][AGT=IPT][MxId=xyzxyzxyz] [xyzxyzxyz.ausprd01.prod.outlook.com 2026-11-13T01:08:15.373Z xyzxyzxyz]
bawolff 6 hours ago [-]
Normally if server down, smtp will wait a little while and try again.
OnceMoreUntoThe 8 hours ago [-]
Nah, no way they're getting rid of their M.C.-Escher-esque 'Forefront' mail-disappearance-machine-that-nobody-in-Microsoft-seems-to-understand-or-control, but, yeah, even their Internet-facing infrastructure seems unhappy:
451 4.3.2 Temporary server error. Please try again later ATTR2 [MI3PEPF00008486.eurprd02.prod.outlook.com 2026-11-12Txxxxxxxxxxxxxx]
suzzer99 8 hours ago [-]
Anything sent to my work or home Outlook email from gmail doesn't bounce back, and anything sent from those is still stuck in the outbox. Microsoft.
mjmas 2 hours ago [-]
It won't actually bounce until either gmail gives up on its retries (which generally is 24 hours) or the outlook server gives an explicit rejection (rather than a temporary error or timeout)
jmspamerton 8 hours ago [-]
[dead]
glass1122 34 minutes ago [-]
[dead]
hassnmauller 8 hours ago [-]
[flagged]
ericfrederich 8 hours ago [-]
Is this related to GTA6 being pushed to consoles a week prior to launch?
Would they ever admit it if that were the case?
I want to believe this is the cause. It'd be especially funny because businesses were already anticipating the productivity loss when GTA6 comes out, but now we have actual productivity loss because of this outage (and it's affecting people who were never going to play the game anyway, lol)
nerdjon 8 hours ago [-]
Why would this be the case? Games preloading has been a thing for many years. This isnt like an MMO launch that there is additional impact when the game is being played.
Sure GTA6 may be a slight uptick in the amount of data being downloaded like any new game being released, but I would wager that the amount of data they are normally pushing out for downloads just on a regular basis (wether new purchases, redownloading games, game pass, etc) is already quite high.
Juvination 8 hours ago [-]
I don't think this is the case, even a little bit.
ratelimitsteve 8 hours ago [-]
I want to believe that cause is that all the world's computing power is dedicated to calculating the incredibly large balance in my checking account but my desire is the only thing I can find to support that hypothesis.
ericfrederich 7 hours ago [-]
An estimated 2.6 million pre-orders for Xbox alone. 2.6 million people downloading 200+ GB is not nothing.
If they roll it out evenly the week prior, then today (1/7th) is just 75 PB. To get that done in a single day would require a sustained 6.9 Tbps.
I'm sure this is all handled by a CDN at some level... but again it's not nothing.
zamadatix 7 hours ago [-]
It is basically nothing once you remember the cloud isn't one computer either. I want to say Azure was already millions of (beefy ass) severs over hundreds of datacenter locations organized into dozens of zones, and not all of those servers in each region are assigned to the same sets of services.
6.9 Tbps to get it done in a single day is even 3x less than the minimum on Steam in the last 48 hours https://store.steampowered.com/stats/content/ and they don't even have GTA VI to distribute.
dorongrinstein 8 hours ago [-]
[flagged]
unethical_ban 8 hours ago [-]
Without digging into it, I don't understand how people can run "multi-cloud" while using the most valuable IaaS that each provider gives - serverless.
Maybe if people are spinning up simple VMs and VPCs they can manage multicloud through a hub, but if you're using the big three for what they're meant for, you're using cloud DBs, API gateways, lambdas, and whatnot. With how much tuning, billing monitoring and specific configuration is needed, it seems like that'd be a hard thing to abstract. Even using an abstraction like Terraform to try to keep out of the weeds takes a lot of work, from what I remember five years ago.
SoftTalker 7 hours ago [-]
Yes, and that's exactly why all the services offer more than simple VM/container hosting. They want you and push you to use all the other stuff that traps you within their moat. The last thing they want is for a competing service to be an easy substitute.
We use Azure at work and it’s wild how much stuff is just manual labor behind the scenes on their platform.
Seems like it’s always on fire somewhere, ripe for burnout and MBOs aren’t aligned with reality from what I hear.
AWS is no different. Everything is on fire all the time, but customers usually don't notice.
New Xbox to cost $1200-$1800.
Gamepass limited to 15 hours a month for Ultimate.
But this is certainly exactly what I'd expect after indiscriminately laying off half your workers, then immediately offering a buyout to make sure any experienced people still standing will find their way to the door.
If this incident, or the difficulty they're having with remediation (update: holy shit, they still haven't even identified the problem), are somehow not caused by any of that, then it's at least a preview of what things are going to be like.
Btw does everyone see how insanely crappy Windows 11 is compared to MacOS
The Status page is not working well, but I managed to subscribe to their RSS feed and it has been sending out updates periodically. I'll post some of them here.
9:02 PM UTC: We've validated an alternative health check method on a subset of affected infrastructure, which allowed the servers in that subset to successfully return to service, and we're now applying this change more broadly. We're also preparing an automated recovery process to continually detect and restart unresponsive servers, to help keep them in service while this change progresses. Additionally, our review of recent changes has identified a small number of configuration updates that we're investigating as potential contributors to the issue.
Previous status: While validating the targeted change, we found that many of the affected servers have become unresponsive, which is preventing us from confirming whether the change is effective. We're restarting these servers so they can come back online with the change in place, and we're applying the same approach to additional affected portions of our infrastructure to validate the results more broadly.
8:52 PM UTC: While validating the targeted change, we found that many of the affected servers have become unresponsive, which is preventing us from confirming whether the change is effective. We're restarting these servers so they can come back online with the change in place, and we're applying the same approach to additional affected portions of our infrastructure to validate the results more broadly.
Previous Status: While we haven't identified any recent changes that correlate with the start of impact, our investigation indicates the failed health checks may originate from a specific component on the affected servers that's responsible for processing them. We're deploying a targeted change to bypass this component on a subset of servers to validate whether this restores availability, and we've taken steps to prevent additional servers from being removed from service while we work to return the affected servers to service. We're also investigating why affected servers are experiencing elevated processor usage while handling requests.
7:52 PM UTC: We've identified that affected servers are failing automated health checks, which is causing them to become unavailable, and this may be related to a recent change to a component involved in processing those checks. We're testing a configuration change on a subset of affected infrastructure to address this and, if successful, will apply it more broadly. In parallel, we've initiated a restart of all affected servers and are validating whether they're successfully returning to service.
7:30 PM UTC: We're aggressively pursuing all pathways to reduce impact. As a precaution, we've paused all deployments and configuration changes while we review recent changes for any correlation to this event. We've determined that the elevated load isn't driven by an overall increase in traffic, but by a large number of servers being unexpectedly unavailable, which is shifting load onto the remaining infrastructure. We're investigating what's causing these servers to become unavailable while we work to safely return them to service.
7:16 PM UTC: We're aggressively pursuing all pathways to reduce impact. As a precaution, we've paused all deployments and configuration changes while we review recent changes for any correlation to this event. We've also moved traffic for some services off the affected infrastructure to further reduce load, with additional traffic moves in progress.
6:59 PM UTC: We've completed redistributing connection traffic across our worldwide infrastructure, and we're moving additional traffic off the affected infrastructure to further reduce load. We're also restarting affected servers to return them to service. In parallel, we're investigating whether the elevated traffic reflects a genuine increase in requests or a shift in how traffic is being routed across our infrastructure.
Perhaps more accurate in our world.
From the 10:11:39 PM UTC update:
Current status: We're continuing to pursue all available pathways to restore service, as our actions taken thus far have not provided the anticipated relief. Our investigation has found that a core process on the affected servers is becoming overwhelmed, which causes the servers to stop responding and contributes to the capacity loss.
To validate a potential recovery approach, we're fully isolating a portion of our service infrastructure, restarting the affected components, and gradually reintroducing traffic to confirm our services remain stable before applying this approach more broadly. We're also analyzing traffic patterns across our entire service infrastructure and evaluating whether isolating traffic within each region may allow affected regions to recover independently.
From the 9:41:46 PM UTC update:
Current status: We're continuing to work through restarting and validating the affected servers to safely return them to service, and we're exploring opportunities to expedite this process. In parallel, we're assessing whether reverting the small number of recent configuration updates we identified could help accelerate recovery.
From the 9:01:14 PM UTC update:
Current status: We've validated an alternative health check method on a subset of affected infrastructure, which allowed the servers in that subset to successfully return to service, and we're now applying this change more broadly. We're also preparing an automated recovery process to continually detect and restart unresponsive servers, to help keep them in service while this change progresses. Additionally, our review of recent changes has identified a small number of configuration updates that we're investigating as potential contributors to the issue.
From the 8:44 PM UTC update:
Current status: While validating the targeted change, we found that many of the affected servers have become unresponsive, which is preventing us from confirming whether the change is effective. We're restarting these servers so they can come back online with the change in place, and we're applying the same approach to additional affected portions of our infrastructure to validate the results more broadly.
Previous Status: While we haven't identified any recent changes that correlate with the start of impact, our investigation indicates the failed health checks may originate from a specific component on the affected servers that's responsible for processing them. We're deploying a targeted change to bypass this component on a subset of servers to validate whether this restores availability, and we've taken steps to prevent additional servers from being removed from service while we work to return the affected servers to service. We're also investigating why affected servers are experiencing elevated processor usage while handling requests.
From the 1:06:17 AM UTC update:
Current status: Our investigation has found that a process on the affected servers responsible for establishing secure connections becomes overwhelmed, causing the servers to stop processing requests. We're preparing a configuration change to address this behavior that can be applied across all affected servers, and we're validating it on a subset of servers before applying it more broadly alongside restarts of the affected infrastructure. In parallel, we've begun reverting a recent network configuration change that correlates with the start of impact and are monitoring to determine whether this contributes to recovery.
The temporary traffic limitations we previously described are now in effect in specific locations. We're monitoring the affected servers in those locations to confirm whether they remain stable while handling a narrower set of connections, after which we'll gradually reintroduce additional traffic.
---
The list of impacted services just grew tremendously.
From the 12:56:09 AM UTC update:
Features of the Microsoft 365 services below that interact with Exchange Online and email services are impacted. Elements of those services that do not depend on interaction with Exchange Online remain functional.
- Exchange Online: Users may be unable to access their mailbox using Outlook on the web, Outlook desktop (MAPI), Exchange ActiveSync (EAS), or REST-based connections, and email delivery may be delayed.
- Microsoft 365 admin center: Admins may be unable to load some admin center pages.
- Microsoft 365 apps: Users may be unable to view their recently used files.
- Microsoft 365 Apps Admin Center: Multiple services including Cloud Update, Cloud Policy, and Inventory are affected.
- Microsoft 365 Education: Features including Assignments, Reading Progress, and Speaker Progress may be impacted.
- Microsoft Copilot 365: Users may be unable to load Copilot Chat or receive responses in the web and desktop apps, and Copilot experiences may be unable to access work data such as email, calendar, and files.
- Microsoft Defender for Office 365: Safe Links may be unable to verify URLs, which may impact users clicking links in Outlook and Teams.
- Microsoft Defender XDR: Users may be unable to load Microsoft Defender portal pages or solution tabs.
- Microsoft Fabric: Users may encounter failures when exporting reports with sensitivity labels to PDF or PowerPoint, or when updating artifacts with sensitivity labels.
- Microsoft Graph: API requests that access Exchange Online data, such as mail, calendar, and contacts, may time out.
- Microsoft Intune
- Microsoft Planner: Users are experiencing Plan and Task operation reliability issues.
- Microsoft Purview: Multiple Purview services including Authorization, auto-labeling, Data Loss Prevention (DLP), Communication Compliance, and Data Classification are impacted, and users may be unable to load Microsoft Purview portal pages.
- Microsoft Teams: Scenarios that depend on Exchange Online, including calendar, meeting scheduling, and creating chats for organizations using Information Barriers, may fail. Core messaging and online meetings remain largely functional.
- Microsoft To Do: The app may not load, and users may see operation failures.
- Power BI: All users may intermittently be unable to export reports with MIP sensitivity labels to PDF in Power BI.
- SharePoint Online: Features that depend on Exchange Online may be impacted. Accessing files stored directly in SharePoint Online remains functional.
- Universal Print: Users may encounter errors submitting print jobs or using admin features.
- Windows 365: Customers who have configured their tenant and Cloud PCs to use Microsoft Purview Customer Key for Windows 365 Cloud PCs may have users who are unable to connect to their Cloud PCs.
Current status: We've reviewed known impact scenarios and have revised the list of impact services and features, which can be viewed in the "more info" section of this communication.
We're continuing our work to isolate the issue affecting server availability, as well as other workstreams.
---
So much for seeming hopeful.
From the 12:08:53 AM UTC update:
Current status: While the servers we've returned to service initially remain healthy, we've observed that they can become unresponsive again once they begin receiving traffic. We're analyzing diagnostic data from these servers with our networking and platform experts to determine what's triggering this behavior.
Building on the protective traffic management measures we've implemented, we're also temporarily limiting which types of traffic can reach the affected infrastructure in specific locations. This will help us determine whether a particular type of request is contributing to the impact and allow us to restore service incrementally. Additionally, we're rebuilding a subset of affected servers using a previous software configuration to validate whether this prevents them from becoming unresponsive, and we'll expand this approach to additional servers if we observe positive results.
---
The 11:05:31 PM UTC update is is the first since the outage began that feels hopeful.
From the 11:06:31 PM UTC update:
Current status: We've begun returning a portion of the affected servers to service, and they're remaining stable so far. We've also shifted traffic and connections to alternative pathways wherever possible and have observed limited recovery in some regions.
We've streamlined our recovery process to expedite returning the remaining affected servers to service. To further support recovery, we're implementing protective traffic management measures in a specific location to reduce load on the affected infrastructure, and we'll expand these measures to additional locations if we observe positive results. As capacity is restored, users may experience a gradual recovery as their clients reconnect to the service.
From the 2:14:51 AM UTC update:
Current status: We're applying infrastructure capacity improvements where possible in an effort to help support traffic in regions coming online. We're continuing to monitor the results of changes applied so we can identify strategic actions that can help to improve service health across all affected regions.
---
Hopefulness returns.
From the 1:50:27 AM UTC update:
Current status: We're seeing positive results with our efforts to restart affected infrastructure. We're actively monitoring the performance of these to verify stability which will support improved availability. Remediation actions will be expanded as we confirm stability with the changes applied.
It's been hours, and the status updates still feel like there is an undertone of frantic panic (understood if you can't determine the root cause). When I think of past major outages, they may have been down for hours but usually folks knew what the root cause was relatively early, even if remediation took some time.
I wouldn't be surprised if AI coding and release pressure was a primary cause, so I really hope the future post-mortem is made publicly available.
Richard Cook wrote about this years ago, and it's always going to be relevant:
https://highscalability.com/designing-for-resiliency-will-be...
https://how.complexsystems.fail/
https://americanliterature.com/author/em-forster/novella/the...
Reddit-level speculation.
I feel for those stuck in there. The outage going on for this long has got to be fucking brutal.
In retrospect, the cause and cure are obvious to everyone.
The problem is amateur hour is the entire industry. Nobody specs anything, nobody tests overload conditions, nobody thinks failure modes. The stuff that every engineer does when building anything else other than software doesn't happen. The orgs are bloated. Too many teams. Too many silos. Incentives are stacked the wrong way. People are motivated to not think about systems but stay in their lane. Responsibilities are split in weird ways.
Yes. Big systems are hard. But we can build, manage and understand big systems with the right approach and people. Even one computer is complicated and many people get stuck trying to debug a piece of software running on one machine. At the end of the day it's not magic. All of humanity is big systems but when I have a power failure the power company is generally able to understand why and resolve it and the water in my tap doesn't stop flowing and the cell towers still work (generators, who would have thunk).
A friend who is really good had to leave Microsoft because of how AI was pushed. Before that he was telling me some other stories that make the culture seem pretty not great. Heard similar stories from other people. It used to be a great place filled with great people until at some point it's just another big company.
My bet is we'll eventually get there. Planes kill people when they fall, software used to not be as dangerous, but it's been a while since it became critical infrastructure.
Given that people on reddit claimed that in the entirety of New Zealand no real estate transactions can be settled, causing cascading failures [1]...
I think this might actually be the incident pushing governments to designate large online services such as MS 365, Gmail et al as critical infrastructure and force companies to massive, yet-unseen levels of transparency and regulation.
The EU is already itching to act ever since the Trump admin forced US Big Tech to terminate service to the ICC judges, and this here was a square own goal for Microsoft.
[1] https://www.reddit.com/r/sysadmin/comments/1xus565/comment/1...
Plenty of internal accounts from previous employees highlight this.
Most of the time we get demonized by resume-driven developers job-hopping every few years (seems like the lack of deep expertise is working out). Today we get to laugh at their expense. These problems were entirely foreseeable and avoidable.
If we could see the memory bits and the CPU internals, I think it would be good candidate, particularly branching and branch prediction.
A brick is very complex. A bearing is complex. A bolt and a nut and a washer are complex. An integrated circuit is complex. We build stuff out of that. Like buildings that don't fall when you sneeze on them.
In software we use duct tape - at best. But that's not just Microsoft, that's pretty much everyone. With maybe few exceptions.
What a great phrase!, can I use that in conversation?
They still don't know why this "core process" is maxing out the processor for no reason.
Site was taken offline, war rooms ensued. 48 hours later, an Ops team from Redmond was flown out. Another three days and they determined it was several legacy servers were going down and the load balancing servers were getting hit so hard, they also failed. It was a cascading effect that killed everything.
CIO was pissed. The company got absolutely lit up on every social media channel for their ineptitude. The CIO vowed it would never happen again, they were upgrading the entire network and infrastructure to avoid it happening again. Fast forward six months to the next major release. Same thing, same panic, same cascading failures because servers were collapsing under the traffic to the website. Another war room, another Ops team flies out. Two days later they figured out it was some buggy code that was causing it.
This after 8 rounds of load balancing testing to the tune of 10X the amount of traffic the site sees on these major release dates without so much as a hiccup.
I left two months later, but the overall feeling was the same - total amateur hour from Microsoft after our company was gifted major discounts to go all in with their hardware & network gear. The company found out the hard way it wasn't a smart decision.
The scary part was this some 20 years ago - seems like not much has changed considering how frequently Github is down now too.
Actual tech write-ups can come later.
You know a lot of talented people who work there also post here, and it's not fair to insinuate that they aren't talented because some vastly complex system somewhere in the world had a bad day.
Stuff happens, things break. That's why lots of nines cost a lot of money. It doesn't automatically mean that anyone is incompetent.
lots of nines means not having significant downtime. we passed 99.99% hours ago, and we're on the way to 99.9%
Almost certainly not the fault of individual people, it's a management failure of course
They transformed Windows from a useful OS to a banal surveillance and advertising platform.
I agree there are many talented people at Microsoft, but I don’t see the professionalism in this incident.
I'm sure it's more complicated than that.... but from my POV, which is extremely limited and informed by what others online tell me.... that this is amateur hour
What would it take for it to be professional hour?
You can't do much to work around nuclear war or a major asteroid strike, but keeping important things working under normal operating conditions should not be a high bar.
looks at the Boeing 787
Put another way, if they didn’t turn it off and on again, I would think less of them because that’s the first thing you should always do unless you know better
We absolutely should have scheduled reboots during off hours. Lessons learned.
we "bounce" the servers when on calls with customers.
Everyone has occasional outages but this is the worst 365 outage our org has seen since starting with M365 in 2019.
https://redmondmag.com/articles/2018/09/11/microsoft-explain...
I think it does _too much_ and the performance is _bad_ but it’s rock-solid for video and audio calls - more than even FaceTime or analog calls.
Using Teams after being used to Slack is like going from VSCode to Eclipse. I’m sure this thing is powerful, but it looks weird, it feels weird, it moves like January molasses, and it has a million features I’m completely uninterested in. I’m sure some people get value from all the baroque. I am not one of them.
(I’d rather be using IRC and Emacs but the world has other ideas.)
Truly, we are in dark times.
"We're also preparing an automated recovery process to continually detect and restart unresponsive servers, to help keep them in service while this change progresses."
https://www.abc.net.au/news/2026-11-13/microsoft-outage-affe...
> Current status: We’re continuing to pursue all available pathways to restore service, as our actions taken thus far have not provided the anticipated relief. Our investigation has found that a core process on the affected servers is becoming overwhelmed, which causes the servers to stop responding and contributes to the capacity loss.
Current status: While the servers we've returned to service initially remain healthy, we've observed that they can become unresponsive again once they begin receiving traffic. We're analyzing diagnostic data from these servers with our networking and platform experts to determine what's triggering this behavior.
Building on the protective traffic management measures we've implemented, we're also temporarily limiting which types of traffic can reach the affected infrastructure in specific locations. This will help us determine whether a particular type of request is contributing to the impact and allow us to restore service incrementally. Additionally, we're rebuilding a subset of affected servers using a previous software configuration to validate whether this prevents them from becoming unresponsive, and we'll expand this approach to additional servers if we observe positive results.
---
The 11:05:31 PM UTC update is is the first since the outage began that feels hopeful.
From the 11:06:31 PM UTC update:
Current status: We've begun returning a portion of the affected servers to service, and they're remaining stable so far. We've also shifted traffic and connections to alternative pathways wherever possible and have observed limited recovery in some regions.
We've streamlined our recovery process to expedite returning the remaining affected servers to service. To further support recovery, we're implementing protective traffic management measures in a specific location to reduce load on the affected infrastructure, and we'll expand these measures to additional locations if we observe positive results. As capacity is restored, users may experience a gradual recovery as their clients reconnect to the service.
---
The 10:11:39 PM UTC update is a little chilling. They don't know why what's happening is happening so they're doing experiments.
I do that with closed-source software when I don't want to bother trying reverse engineer it, but these guys have the code! Yikes.
I recognize that even w/ the code available there are likely behaviors that aren't likely to be anticipated because of the scale of traffic. Complex systems fail in complex and non-intuitive ways.
From the 10:11:39 PM UTC update:
Current status: We're continuing to pursue all available pathways to restore service, as our actions taken thus far have not provided the anticipated relief. Our investigation has found that a core process on the affected servers is becoming overwhelmed, which causes the servers to stop responding and contributes to the capacity loss.
To validate a potential recovery approach, we're fully isolating a portion of our service infrastructure, restarting the affected components, and gradually reintroducing traffic to confirm our services remain stable before applying this approach more broadly. We're also analyzing traffic patterns across our entire service infrastructure and evaluating whether isolating traffic within each region may allow affected regions to recover independently.
I mean it sort of does if you know the whole system inside and out and have the ability to reason about it, but just having the code for each component doesn't by itself solve the real problem of understanding
But maybe don't understand it because they didn't write it themselves?
2. Art imitate Life
3. Goto 1
https://youtu.be/3TNpOD6bov8
For reasons that I cannot fathom, Microsoft wants to act as the relay to my self-hosted e-mail account, if I add the account in Outlook.
I can conjecture some reasons. Datamining would be high on the list, since MS can read your mail and thus market to you more effectively. Governments might also be happier if MS can provide them with one-stop access (like Prism) to otherwise self-hosted e-mail accounts proxied through a central server.
BUT, more likely, it's some sort of "check in" or "dial home" feature that isn't working. Or plausibly, maybe they count "local user checked email" and their analytics endpoint is down.
Regardless, it's definitely surprising that a traditional desktop application would be blocking when only polling from local/trusted data sources.
It is normally probably more reliable; except when their server is down.
I'm being an ass in saying this, because I'm aware human motivation is a beautiful kaleidoscope of variables, but I would have guessed at the venn diagram of 'those willing to compromise and use a Microsoft product' is mutually exclusive with that of the 'I'm hardcore enough to self-host email'.
I will say, however, that Outlook used to be one of the better desktop email programs, so I'll grant some legacy lenience.
In a corporate environment, I don't think this would be that unusual. But yes, it's hard to imagine someone would do that in a personal setup.
Great oldschool UI coupled with support for all the modern protocols, and consistently updated: https://www.rimarts.jp/downloads/B2/Readme-e.txt
I use Gmail's web interface for my Gmail account (which, I'll admit, makes my little rant above more than a little hypocritical)
And in my self hosted email setup I use RoundCube (web interface).
Na, that would simply be the kaleidoscope part you mentioned, wouldn't it? For what it's worth, I found the combination just as peculiar as you did.
Unfortunately, being someone who speaks up nowadays has been portrayed as being an asshole.
How dare you!?
I’m pretty certain because my company’s phishing-test emails skip my filters (I assume admins have a “skip filters” toggle). Which, hilariously, makes it even easier to ensure there’s a place that contains only phishing test emails (filter all not on allow-list of domains to one place, filter all on that list to another, ta-da, the main inbox now contains only phishing test emails)
I assume they do the same thing with non-MS email if you use their client, rather than maintain two filter implementations.
All around, yeah, wow, super gross.
Biggest data heist and most obvious US intelligence operation in human history. I can't fathom why AVs don't treat "Outlook New" as info stealer malware.
One of GitHub's services will be down again by next week.
Any other service that interacts with Outlook in some way will create downstream effects in other services expecting Outlook to work normally.
If your org is using Intune or Defender EDR then you probably also are impacted.
The Y axes are entirely independent, and so when there's an outage for Microsoft Outlook (which peaked at ~130k reports) and you get a negligable spike in pepole who get confused and report gmail (~1k reports), the graphs all look correlated and connected.
They need to show the graphs with an absolute y-axis set by the worst outage, or scale the graphs by estimated average daily users or something.
Or, rather, it's acting like it usually does.
The worst part about Outlook is the broken search. In the world of infinitely smart AI, the e-mail search in Outlook is embarrassingly stupid.
I just assumed it was Teams being Teams.
Back on Window Mobile 6.1.
Battery life usage quickly improved. :)
DDOS?
First, you have every client retrying connections, whether automated or manual retries. This generates extra load over normal.
In addition, whichever systems come online first often get a ton of traffic sent to them, which can immediately bring them back down if you don't have ways to prevent it. You will get a wave of systems coming back online, a ton of traffic hitting them, and then falling over again.
Now, I am sure Microsoft is used to this and has procedures and systems in place to mitigate the issues (I know we did).
However, sometimes if the right combination of services are failing, even the strategies you had to avoid this sort of failure can themselves fail.
However, in other cases it can be more complicated. For example, if the requesters are other internal services, they might not be behind a single gateway (although you should then have other controls you can use). Other times, it is not easy to block the RIGHT requests, because it is hard to distinguish which backend services will end up being hit by a request (a single request can trigger multiple calls to multiple services).
There is also the problem of catch up - if a system in a stable state is close to max capacity, it might never have the capacity to recover from a failure even if the entire system is restored. For example, imagine a log processing system that can just barely handle the normal peak load; if you have a failure where your producers locally store logs until they can send them upstream, you will have to clear that backlog, which requires additional capacity beyond a full steady state, which you might not have.
For Copilot.
This seems to be the smoking gnu, now what would it be ...
(I've recently been revisiting the X-Files. Can recommend.)
I saw a frog smoking on The Little Rascals once, but never a gnu.
EDIT: Seems like now they confirmed that: "We've identified that affected servers are failing automated health checks, which is causing them to become unavailable, and this may be related to a recent change to a component involved in processing those checks. We're testing a configuration change on a subset of affected infrastructure to address this and, if successful, will apply it more broadly. In parallel, we've initiated a restart of all affected servers and are validating whether they're successfully returning to service."
@MSFT365Status
16m
We've not identified any recent change correlated with the start of impact; however, our investigation indicates the failed health checks may originate from a specific component that's responsible for processing server health. We've taken steps to prevent additional servers from being impacted while we work to return the affected servers to service. More info under EX1550543 and MO1550821 in the Microsoft 365 admin center, as well as status.cloud.microsoft if the admin center is not available to you.
> Previous status: We're continuing to work through restarting and validating the affected servers to safely return them to service, and we're exploring opportunities to expedite this process. In parallel, we're assessing whether reverting the small number of recent configuration updates we identified could help accelerate recovery.
If failures started at some point in time then something must have changed just prior to that. If no changes in traffic were detected (no DDOS) then it must have been a change, be it code or config. Duuuh.
Not at all, they could have simply hit an unexpected behavior for their infra they hadn't seen, tested, or expected before. Global scale infra gets very, very complex, and tbh this is Microsoft... they're not very good at it.
More often than not, it's DNS :)
Oh... Oh I have news for you :)
Edit: Oh, there it is.
Lamest tech company imo. Why dont you lay off some more staff, Satya, maybe that will fix the problem.
ThousandEyes Internet Insights offers a public outage map at https://www.thousandeyes.com/outages/ that is MASSIVELY more accurate (and faster to detect) than Down Detector because its based on actual telemetry and not on crowdsourcing.
They indicate sharepoint is getting blown out... again.
There will be a more detailed message when we finish the SII on our status page at the link I previously provided, but the short version is we took way more traffic than normal to our platform and the related workers were running out of memory.
No. It requires an account now, and they did away with the recent posts firehose.
I often know about outages dozens of minutes in advance of it being posted on official channels (or in the case of AWS, three days later...).
So what if the resolution is low because people can't log in to Fortnite and an Azure outage gets shown as an Epic Games outage. Doesn't matter. The live feed is incredibly reassuring from a "down for anyone else this instant, before official status page has a chance to see it?" Saves me so much time knowing whether it's a me-or-them issue.
And with Google is people complaining Chrome doesn't load their "pages". I bet they mean outlook.
That's a bold claim to make on HN.
https://status.cloud.microsoft/m365
Azure status does not document anything:
https://azure.status.microsoft/en-us/status
All that said... Stuff on my home PC started getting various DNS issues around the time my email started acting weird. Hard to say how widespread it is. It may be bigger than Microsoft even.
Interesting: there's a huge correlation with everybody beginning to use LLMs.
And we dismiss it as being non-causal by saying it all started going downhill before the advent of LLMs?
I don't buy it. I think we're seeing the results of what happens when you hand the keys of the kingdom to LLMs.
It's been full speed downhill for many years, and the failure modes are pretty distinct.
Current industry best practice is to use LLMs improperly.
That doesn’t mean the code/result doesn’t still need to meet our engineering standards.
One of the best ways to deal with doing something the wrong way is to get rid of all the people who know how to do it correctly. That way no one knows any better, so no one has to confront the fact they're actually doing it wrong.
That is a software issue.
Service quality is in better hands after the layoffs, at least that’s what investors are led to believe.
The number of times it has spit out a python program where a much simpler bash script would do, is too many to count.
I have a pet theory that if you don't know what you really want, and aren't skilled enough to understand what you got, infra code is THE WORST.
Anecdote: I wanted a simple change to an ansible playbook today. GPT-6-Astra/codex came up with the most convoluted, brittle solution possible the first time, even though it was extending and already clear, idiomatic playbook. It took some amount of offensive and critical prompting to get it to use ansible's built in modules and tooling to do the same job - and do so with nearly 80% less configuration and like 1000% more human readability.
Some part of me harbours a sneaking suspicion that the "throw everything at it" ethos is by design, not as a defensive measure against crappy code, but as a way to ensure there's so much code and it's so convoluted orgs and operators (or developers) are stuck/addicted.
It's still a force multiplier - but it's by no means driverless, especially for existing/already-running infra.
(they also used to have local admin rights on all of their client computers until one employee got phished and literally gave away their personal ID card via a webcam)
In any case, guess who got to implement all the relevant playbooks for installing software and what tool they used for it? Our company literally handed the job to an intern who had no IT background whatsoever. I say this not to be mean, he was legitimately just at our company because a government program required him to do an internship.
He used Deepseek (our company does not believe in purchasing LLM subscriptions, at least not for us normal employees) to write literally all of the playbooks for any kind of software that mattered to our clients.
Now, as I said before, I am only an apprentice, and I am certainly not a good programmer by any means, but I read through parts of the Ansible documentation and had worked with it before, and I understood enough to at least be able to roughly judge whether a playbook made sense or was just way too complicated and convoluted.
I had to help this intern out multiple times, because he hadn't yet understood that, in order to write a functioning playbook, you first had to somewhat understand how to deploy a piece of software on a single test machine using CMD or Powershell syntax with an according .exe or .msi.
He simply prompted, pasted the playbook and then reprompted with the error codes from the failed playbook until it somehow worked or until I had to help him out.
This took place in December or November of last year and the playbooks are mostly still pretty shitty if I am being honest.
I did take away an important lesson from this "episode", though: LLMs can be very usefuly for writing code, or in this case, Ansible playbooks, but only if you have a somewhat decent and basic grasp of how software installation works via CMD/powershell and if you took the time to read through and learn about at least the basic Ansible concepts.
Since I am only an apprentice (on paper and in terms of my knowledge and how I get paid, not in terms of how much I am expected to earn for the company), I have painfully experienced how important it is to possess a certain level of understanding about the infra-related code and scripts that you would like LLMs to write for you, because otherwise it very quickly spirals out of control and starts throwing everything at the problem, just as you described.
SignDrop is an unpatched Windows zero-day local privilege escalation (LPE) exploit that allows a standard user to gain NT AUTHORITY\SYSTEM privileges by abusing the Common Log File System (CLFS) kernel driver (clfs.sys). Security researcher Abdelhamid Naceri (also known as MSNightmare or Nightmare Eclipse) published a detailed technical write-up and functional proof-of-concept (PoC) code on GitHub on November 10, 2026.
> We're investigating reports of users experiencing issues accessing Exchange Online mailboxes through multiple connection methods in the Americas. More details can be found in the Microsoft 365 admin center under EX1550543.
Personally I kinda enjoy these outages…
Status page is down. Much of our org is halted.
Fingers crossed that Azure isn't impacted (seems fine so far)
(https://www.geekwire.com/2026/microsoft-makes-new-voluntary-...)
Just clarifying this in general, when a company makes a voluntary offer to some segment of employees, especially at this size, it is a statistics game of how many people accept the buy out.
They did not make the offer to 25% of the employees to downsize by 25%, they did it to downsize by 5% (or something like that), and then made the offer based on how many they expect to accept. This is specifically covered in this line of the geekwire article:
> Microsoft didn’t provide a projection for how many employees will accept, but the company told GeekWire it anticipates a much smaller share than the roughly 30% who took its earlier voluntary retirement offer, which went to longtime employees more likely to take the deal.
They probably expect 10-20% of those offered to take it, which would result in a 2.5-5% total downsizing.
Or is this just a gesture of goodwill to avoid people jumping the ship on their own?
And the other part is the "goodwill" you mention - if you jettison people for nothin en masse, word gets out, and make it hard to hire in the future
People who aren't motivated and/or don't back themselves to get another job will stay (i.e. probably the people you want to get rid of)
Claims do affect the rate they pay next year, but the company isn’t “on the hook” for you personally, at least not any more than they are for your social security income.
And everybody has spent a year letting AI write code. Correlation is not causation, but I wonder...
The page is showing a "Service Degredation" for "Microsoft 365 (Business or Enterprise)"
Updates below in reverse chronological order:
From the Thurdsay, November 12, 2026 at 7:30:36 PM UTC update:
Current status: We're aggressively pursuing all pathways to reduce impact. As a precaution, we've paused all deployments and configuration changes while we review recent changes for any correlation to this event. We've determined that the elevated load isn't driven by an overall increase in traffic, but by a large number of servers being unexpectedly unavailable, which is shifting load onto the remaining infrastructure. We're investigating what's causing these servers to become unavailable while we work to safely return them to service.
---
From the Thurdsay, November 12, 2026 at 7:16:03 PM UTC update:
Current status: We're aggressively pursuing all pathways to reduce impact. As a precaution, we've paused all deployments and configuration changes while we review recent changes for any correlation to this event. We've also moved traffic for some services off the affected infrastructure to further reduce load, with additional traffic moves in progress.
---
From the Thurdsay, November 12, 2026 at 6:59:24 PM UTC update:
Current status: We've completed redistributing connection traffic across our worldwide infrastructure, and we're moving additional traffic off the affected infrastructure to further reduce load. We're also restarting affected servers to return them to service. In parallel, we're investigating whether the elevated traffic reflects a genuine increase in requests or a shift in how traffic is being routed across our infrastructure.
---
From the Thursday, November 12, 2026 at 6:32:25 PM UTC update:
Title: Users may be unable to connect to multiple Microsoft 365 services
User impact: Users may be unable to connect to multiple Microsoft 365 services.
More info: Potentially impacted services include:
- Exchange Online
- Microsoft Teams
- SharePoint Online
- Microsoft Copilot 365
- Microsoft Defender XDR
Current status: We’re investigating an issue impacting multiple Microsoft 365 services and we've determined the affected infrastructure is experiencing elevated load across multiple regions, which is likely contributing to the connectivity errors. We're restarting and repairing impacted portions of this infrastructure to restore capacity, and reviewing an unexpected increase in a specific type of service traffic to determine whether it's contributing to the elevated load. In parallel, we're evaluating options to redirect traffic to alternate infrastructure to expedite relief.
Scope of impact: This issue may affect users attempting to access the affected Microsoft 365 services.
---
From the Thursday, November 12, 2026 at 6:10:02 PM UTC update:
Scope of impact: This issue may affect users attempting to access the affected Microsoft 365 services.
User impact: Users may be unable to connect to their mailboxes in Exchange Online through multiple connection methods.
More info: Potentially impacted connection methods may include:
- Outlook on the web
- Messaging API (MAPI)
- Exchange ActiveSync (EAS)
- Representational State Transfer (REST) API
Current status: We’ve identified connectivity errors occurring on a portion of Exchange Online infrastructure which receives and processes users’ access requests. We’ve collected networking logs to review to further investigate the source of these errors and confirm our next steps.
Scope of impact: This issue may affect users attempting to access their mailbox in Exchange Online using the affected connection methods.
Next update by: Thursday, November 12, 2026, at 8:00 PM UTC
---
Aside: What with the nature of the Teams, OneDrive, etc, products being Sharepoint and Exchange in a trenchcoat this is probably creating quite a mess for services besides Exchange Online, too.
Edit: Looks like the status page needs a status page.
Edit: Verified w/ multiple other parties that Teams and OneDrive are having issues, too. Interestingly, Microsoft 365 Family services are working fine. That makes me wonder if that product is running on separate infrastructure (or just hasn't caught fire yet).
Edit: The RSS feed seems to load fairly quickly: https://status.cloud.microsoft/api/feed/mac
What MS has posted so far:
>> Microsoft 365 Status >> @MSFT365Status >> We're investigating reports of users experiencing issues accessing Exchange Online mailboxes through multiple connection methods in the Americas. More details can be found in the Microsoft 365 admin center under EX1550543. 12:54 PM · Nov 12, 2026
>> Microsoft 365 Status >> @MSFT365Status >> While we're working to mitigate impact, we've confirmed the issue may affect Microsoft 365 service connectivity for users in any region, more information will be provided in the Microsoft 365 admin center under EX1550543.
>> Microsoft 365 Status >> @MSFT365Status >> We've determined a portion of infrastructure is experiencing elevated load across multiple regions, which is likely contributing to the connectivity errors. We're providing more information under EX1550543 and MO1550821 in the Microsoft 365 admin center, as well as https://status.cloud.microsoft
>> Microsoft 365 Status >> @MSFT365Status >> 20m >> We've begun redistributing connection traffic across our broader worldwide infrastructure to help balance the load. More info under EX1550543 and MO1550821 in the Microsoft 365 admin center, as well as https://status.cloud.microsoft if the admin center is not available to you.
[0] https://x.com/MSFT365Status/status/2120938282204688468
The biggest site I work for, who went M365 hybrid last year, has incurred more Exchange downtime today than they had all of last year with their on-prem Exchange servers.
I mean, either roll back your update or there is a malicious actor involved.
Sure, the program on your desktop computer is fine, but the Outlook app on your phone routes everything through MSFT's servers, so you may be impacted. Outlook.com is running EWS, so it would be down as well.
And finally, if you're talking to a layperson, they're not going to know what Exchange is. They just know they opened Outlook and it isn't working.
"School staff are currently unable to receive or send emails due to a nationwide Microsoft Outlook outage. We hope that the issue will be resolved soon and will send an update when we have more information."
- live.com is down
- onedrive loads the UI, but all of my contents are missing
- MS Edge sync is down (503s )
Yup, major-major outage.
In other words, a day that ends in "y".
My company was recently acquired by a company that uses Office, so I'm using Microsoft products for the first time in 15 years. It's absolutely APPALLING how bad they are. I assumed they got fairly usable in the last 15 years, but it seems they've gone backwards.
For all my gripes about GSuite, ... I don't know how businesses can justify using software this bad.
O365 is Microsoft Office shoehorned into running in a web browser.
It's funny how text-only niche forum that looks like it's running on RPi stuffed in someone's closet is routinely used to discuss active outages in multimillion/billion corps.
Hope this entertains you while we can't get to our emails.
I think it's safe to say, if the server is unresponsive, the change is not effective.
My employer is very very close to MSFT. Both my employer and MSFT have been aggressively moving jobs to India, and firing people in the US. It's hard not to feel some schadenfreude.
That sounds bad
While I don't expect this to be the case, I'm emotionally prepared for this to be some sort of "dicks out for Harambe"-style action protesting JH's departure from Nvidia, on a larger and more technical scale.
Edit: Typo.
I have a teams app with Outlook integration and received a bunch of 502s to smba.trafficmanager.net and a my logs are riddled with "RestError Temporary connection issue when contacting service. Please wait before retrying request"
I will say this, having apps deployed on Slack and Teams, Slack rarely has issues (an exponential backoff is enough to simply never worry about my Slack integration). With Teams I have mysterious failures to debug at least 2x a year - and they're often not very transparent about them (often my first instinct is to search Twitter and find people complaining about the same thing).
After that they will tackle email servers.
Because, well, once upon a time there was Exchange Server and MAPI (the "Messaging API"), and that actually worked pretty well, and pretty much everyone started using it (or one of its EWS/ActiveSync offshoots), but then the Internet happened and ruined everything, but we got MAPI-over-HTTPS eventually, which by that time had grown so many warts that nobody at Microsoft really understood it anymore, so that got deprecated, even though, well, everyone and everything still uses it.
The new hotness was Azure-slash-Office-365, and that's where the "new Outlook" web interface lived and (sort-of) continues to live, and the on-prem customers got stuck with the "Exchange Subscription Edition", which was basically Exchange 2019 plus security hotfixes at extortionate rates, but with none of the new functionality.
And, since MAPI continued to be really, really hard even in the age of Copilot, access from the new hotness to the deprecated on-prem servers was implemented via IMAP (LOL), but force-proxied via the outlook.com cloud/clownshow.
Which is why, right now, many Microsoft customers that thought they were self-hosting their email are entirely down. Lessons might be learned here, but I sort-of doubt it...
Sure, "New Outlook" on Windows is still entirely broken (including stuff like, oh, re-ordering mailboxes), but on Mac it's actually quite usable, and even allows you to switch off the IMAP proxying.
Also laughing at our ops guys who are now absolutely helpless and have people phoning them. I warned about this on our risk call and it was seen as low probability low risk. I hope it's down for a few hours :)
A fictional story about AI could be that it rolled out its new document, spreadsheet, slides, etc. apps, knows the IPO is coming up and decided to make a lot of opportunities for people to sign up for their new release by DDosing their competitor.
> [Home Affairs Minister Tony Burke] says the government has activated a digital resilience agreement with Microsoft today to ensure Australia gets priority:
> "That agreement has in fact been activated today to make sure that we get appropriate prioritisation of Australian interests in this across everything.
It's a little pathetic-seeming to plead for special treatment from Microsoft. I get Australia is an important strategic alliance* for the US, but unless these resiliency agreements are rare... what meaningful prioritisation of "Australian interests" can we expect from MS?
*a useful staging ground to field subs into the South Pacific and Indian Ocean
So.. > Be the last standing SRE at Micro$slop > Your mentors have all left, taking the exit payout, now your days are the best b/c you answer to nobody and can hide in your mountain of markdown. > You knew each of the 7 different products named CoPilot are used to monitor your performance. So you set them all up talking to eachother, burning tokens and API limits. > You are team top performer now, you get all the important decisions now > You walk in the office this morning, CISA is sitting at your desk, with CVE-2026-124194 on your whiteboard > 'You have a patch for this right, China is actively exploiting our Defense primes..'? > Oh no, I haven't landed a patch in prod since my mentor left (because he actually did most of the work) > Oh well, must be close enough, LGTM, CISA seems upset so better just yeet this to prod... > ...
inb4 this is more accurate than anyone wants it to be.
None of the admin ran shops are down 1 day per year.
So in 10h
Now? I dont care Im logged off.
[1] https://github.com/customer-terms/github-online-services-sla
7:52 PM UTC: We've identified that affected servers are failing automated health checks, which is causing them to become unavailable, and this may be related to a recent change to a component involved in processing those checks. We're testing a configuration change on a subset of affected infrastructure to address this and, if successful, will apply it more broadly. In parallel, we've initiated a restart of all affected servers and are validating whether they're successfully returning to service.
7:30 PM UTC: We're aggressively pursuing all pathways to reduce impact. As a precaution, we've paused all deployments and configuration changes while we review recent changes for any correlation to this event. We've determined that the elevated load isn't driven by an overall increase in traffic, but by a large number of servers being unexpectedly unavailable, which is shifting load onto the remaining infrastructure. We're investigating what's causing these servers to become unavailable while we work to safely return them to service.
7:16 PM UTC: We're aggressively pursuing all pathways to reduce impact. As a precaution, we've paused all deployments and configuration changes while we review recent changes for any correlation to this event. We've also moved traffic for some services off the affected infrastructure to further reduce load, with additional traffic moves in progress.
6:59 PM UTC: We've completed redistributing connection traffic across our worldwide infrastructure, and we're moving additional traffic off the affected infrastructure to further reduce load. We're also restarting affected servers to return them to service. In parallel, we're investigating whether the elevated traffic reflects a genuine increase in requests or a shift in how traffic is being routed across our infrastructure.
I'd rather Microsoft host it for me than have to maintain a team of people with Microsoft Exchange skills running either on a cloud environment (similar issues) or on my own infrastructure.
I for one don't miss the days of Exchange going offline or having issues and being the person having to try figure it out.
Microsoft has much better reliability than MOST companies.
When your outsourced system to Microsoft goes down every few months your boss can’t yell at you
Outsourcing is about shifting blame elsewhere
> I dislike Microsoft as much as the next guy, but many businesses impacted by these outages are completely innocent and don't deserve these consequences.
At no point is "keeping people alive" mentioned. Businesses use email for everything. Businesses usually don't care about where their email server is hosted. They just want to send emails. They shouldn't have to care either. I don't put blame on a business that happened to choose Outlook, one of the largest email providers, to host their email.
451 4.3.2 Temporary server error. Please try again later ATTR2 [MI3PEPF00008486.eurprd02.prod.outlook.com 2026-11-12Txxxxxxxxxxxxxx]
I want to believe this is the cause. It'd be especially funny because businesses were already anticipating the productivity loss when GTA6 comes out, but now we have actual productivity loss because of this outage (and it's affecting people who were never going to play the game anyway, lol)
Sure GTA6 may be a slight uptick in the amount of data being downloaded like any new game being released, but I would wager that the amount of data they are normally pushing out for downloads just on a regular basis (wether new purchases, redownloading games, game pass, etc) is already quite high.
If they roll it out evenly the week prior, then today (1/7th) is just 75 PB. To get that done in a single day would require a sustained 6.9 Tbps.
I'm sure this is all handled by a CDN at some level... but again it's not nothing.
6.9 Tbps to get it done in a single day is even 3x less than the minimum on Steam in the last 48 hours https://store.steampowered.com/stats/content/ and they don't even have GTA VI to distribute.
Maybe if people are spinning up simple VMs and VPCs they can manage multicloud through a hub, but if you're using the big three for what they're meant for, you're using cloud DBs, API gateways, lambdas, and whatnot. With how much tuning, billing monitoring and specific configuration is needed, it seems like that'd be a hard thing to abstract. Even using an abstraction like Terraform to try to keep out of the weeds takes a lot of work, from what I remember five years ago.