Chances are there&#x27;s some cyclical dependencies. These can creep up unnoticed without regular testing, which is not really possible at AWS scale unless they want to have regular planned outages to guard against that.

It&#x27;s plausible that Amazon removes unhealthy servers from all round-robins including DNS. If all servers are unhealthy, no DNS.Alternatively, perhaps their DNS service stopped responding to queries or even removed itself from BGP. It&#x27;s possible for us mere mortals to tell which of these is the case.

Looks like maybe a DNS issue? 
<a href="https:&#x2F;&#x2F;www.whatsmydns.net&#x2F;#A&#x2F;dynamodb.us-east-1.amazonaws.com" rel="nofollow">https:&#x2F;&#x2F;www.whatsmydns.net&#x2F;#A&#x2F;dynamodb.us-east-1.amazonaws.c...</a>Resolves to nothing.

Yep, first alert for us fired @ 2025-10-20T06:55:16Z

Yes, we&#x27;re seeing issues with Dynamo, and potentially other AWS services.Appears to have happened within the last 10-15 minutes.

The RDS proxy for our postgres DB went down.

We&#x27;re also seeing issues with Lambda and RDS proxy endpoint.

We changed our db connection settings to go direct to the db and that&#x27;s working.
Try taking the proxy out the loop

We&#x27;re seeing issues with RDS proxy.  Wouldn&#x27;t be surprised if a DNS issue was the cause, but who knows, will wait for the postmortem.

We&#x27;re seeing issues with multiple AWS services <a href="https:&#x2F;&#x2F;health.aws.amazon.com&#x2F;health&#x2F;status" rel="nofollow">https:&#x2F;&#x2F;health.aws.amazon.com&#x2F;health&#x2F;status</a>

<a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=45640754">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=45640754</a>

This is widespread. ECR, EC2, Secrets Manager, Dynamo, IAM are what I&#x27;ve personally seen down.

It’s super broken for me. Random threads no longer appear.

It’s acting up for me but wondering if it’s unrelated. Imagines failing to post and threads acting strange.

Self hosting is golden. Sadly we already feel like we have too many services for our company&#x27;s size, and the sensitivity of vulnerabilities in customer systems precludes unencrypted comms. IRC+TLS could be used but we also regularly send screenshots and such in self-destructing messages (not that an attacker couldn&#x27;t disable that, but to avoid there being a giant archive when we do have some sort of compromise), so we&#x27;d rather fall back to something with a similar featuresetAs a degraded-state fallback, email is what we&#x27;re using now (we have our clients configured to encrypt with PGP by default, we use it for any internal email and also when the customer has PGP so everyone knows how to use that)

Our last resort fallbacks are channels on different IRC servers. They always hold.

Well, at least I now know that my Belgian university&#x27;s Blackboard environment is running on AWS :)

&gt; We&#x27;re having fun figuring out how to communicate amongst colleagues now!When Slack was down we used... google... google mail? chat. When you go to gmail there is actually a chat app on the left.

Signal is down from several vantage points and accounts in Europe, I&#x27;d guess because of this dependence on Amazon overseasWe&#x27;re having fun figuring out how to communicate amongst colleagues now! It&#x27;s when it&#x27;s gone when you realise your dependence

Couple of weeks or months ago the front page was saying how us-east-1 instability was a thing of the past due to &lt;whatever chang of architecture&gt;.

Yup, never add anything new to us-east-1. There is never a good reason to willingly use that region.

&gt;It didn&#x27;t help that some services had their deploy trains (pipelines in amazon lingo) of ~3 weeks, with us-east-1 being the last one.oof, so you&#x27;re saying this outage could be cause by a change merged 3 weeks ago?

When I was there at aws (left about a decade ago), us-east-1 was considered least stable, because it was the biggest.I.e. some bottle-necks in new code appearing only _after_ you&#x27;ve deployed there, which is of course too late.It didn&#x27;t help that some services had their deploy trains (pipelines in amazon lingo) of ~3 weeks, with us-east-1 being the last one.I bet the situation hasn&#x27;t changed much since.

Couple of years ago us-east was considered the least stable region here on HN due to its age. Is that still a thing?

And ring! Don’t know why the chime needs an AWS connection. That was surprising.

Can confirm. I was trying to send the newsletter (with SES) and it didn&#x27;t work. I was thinking my local boto3 was old, but I figured I should check HN just in case.

I cannot create a support ticket with AWS as well.

zoom unable to send messages now as well.

Confirmed.&gt; Based on our investigation, the issue appears to be related to DNS resolution of the DynamoDB API endpoint in US-EAST-1.

There&#x27;s also dynamodb-fips.us-east-1.amazonaws.com if the main endpoint is having trouble. I&#x27;m not sure if this record was affected the same way during this event.

Can&#x27;t resolve any records for dynamodb.us-east-1.amazonaws.comHowever, if you desperately need to access it you can force resolve it to 3.218.182.212. Seems to work for me. DNS through HNcurl -v --resolve &quot;dynamodb.us-east-1.amazonaws.com:443:3.218.182.212&quot; <a href="https:&#x2F;&#x2F;dynamodb.us-east-1.amazonaws.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;dynamodb.us-east-1.amazonaws.com&#x2F;</a>

Cant even login via the AWS access portal.

Can&#x27;t even get STS tokens. RDS Proxy is down, SQS, Managed Kafka.

Amazon.ca is degraded, some product pages load but can&#x27;t see prices. Amusing.

DockerHub shows full outage: <a href="https:&#x2F;&#x2F;www.dockerstatus.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.dockerstatus.com&#x2F;</a>

docker hub or github cache internal maybe is affected:Booting builder
 &#x2F;usr&#x2F;bin&#x2F;docker buildx inspect --bootstrap --builder builder-1c223ad9-e21b-41c7-a28e-69eea59c8dac
 #1 [internal] booting buildkit
 #1 pulling image moby&#x2F;buildkit:buildx-stable-1
 #1 pulling image moby&#x2F;buildkit:buildx-stable-1 9.6s done
 #1 ERROR: received unexpected HTTP status: 500 Internal Server Error
 ------
 &gt; [internal] booting buildkit:
 ------
 ERROR: received unexpected HTTP status: 500 Internal Server Error

I&#x27;m afraid you missed the emoticon at the end of the sentence.A `;)` is normally understoond to mean the author isn&#x27;t entirely serious, and is making light of something or other.Perhaps you American downvoters were on call and woke up with a fright, and perhaps too much time to browse Hacker News. ;)

Could you give us that uptime, in number ?

So much for the peeps claiming amazing Cloud uptime ;)

1. Competitors are not any better, or worse2. Trusted brand

Does any host provide more compensation than refund for downtime?

Last time I checked the standard SLA is actually 99 % and the only compensation you get for downtime is a refund. Which is why I don&#x27;t use AWS for anything mission critical.

They guarantee the dashboard will be green 99.999% of the time

us-east-1 the worst region for availability

It&#x27;s usually true if you arent in US-East-1 which is widely known to be the least reliable location. Theres no reason anyone should be deploying anything new to it these days.

Actual multi-region replication is hard and forces you to think about complicated things like the CAP theorem&#x2F;etc. It&#x27;s easier to pretend AWS magically solves that problem for you.Which is actually totally fine for the vast majority of things, otherwise there would be actual commercial pressures to make sure systems are resilient to such outages.

You are supposed to build multi regional services if you need higher resilience.

You can. You just need to do the work to make it work. That&#x27;s the bit where everyone but Netflix and Amazon fail.

no, it&#x27;s probably Route 53, touted as having &quot;100% availability&quot; (<a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Amazon_Route_53" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Amazon_Route_53</a>)

the highest availability service i think is S3 at 4 ninesyou might be thinking of durability for s3 which is 11 nines, and i&#x27;ve never heard of anyone losing an object yet

I can tell you from personal experience that improving&#x2F;maintaining uptime (by doing root cause analysis, writing correction of error reports, going through application security reviews, writing&#x2F;reviewing design docs for safely deploying changes, working on operational improvements to services) probably takes up a majority of most AWS engineers&#x27; time. I&#x27;m genuinely curious what you are basing the opinion &quot;Delivery of those nines is not a priority&quot; off of.

The point of AWS is to promise you the nines and make you feel good about it. Your typical &quot;growth &amp; engagement&quot; startup CEO can feel good and make his own customers feel good about how his startup will survive a nuclear war.Delivery of those nines is not a priority. Not for the cloud provider - because they can just lie their way out of it by not updating their status page - and even when they don&#x27;t, they merely have to forego some of their insane profit margin for a couple hours in compensation. No provider will actually put their ass on the line and offer you anything beyond their own profit margin.This is not an issue for most cloud clients either because they keep putting up with it (lying on the status page wouldn&#x27;t be a thing if clients cared) - the unspoken truth is that nobody cares that your &quot;growth &amp; engagement&quot; thing is down for an hour or so, so nobody makes anything more than a ceremonial stink about it (chances are, the thing goes down&#x2F;misbehaves regularly anyway every time the new JS vibecoder or &quot;AI employee&quot; deploys something, regardless of cloud reliability).Things where nines actually matter will generally invest in self-managed disaster recovery plans that are regularly tested. This also means it will generally be built differently and far away from your typical &quot;cloud native&quot; dumpster fire. Depending on how many nines you actually need (aka what&#x27;s the cost of not meeting that target - which directly controls how much budget you have to ensure you always meet it), you might be building something closer to aircraft avionics with the same development practices, testing and rigor.

Wasn&#x27;t the point why AWS is so much premium that you will always get at least 6 nines if not more in availability?

Why after all these years is us-east-1 such a SPOF?

Check the URL, we had an issue a couple of years ago with the Workspaces. US East was down but all of our stuff was in EU.Turns out the default URL was hardcoded to use the us east interface and just by going to workspaces and then editing your URL to be the local region got everyone working again.Unless you mean nothing is working for you at the moment.

Doesn&#x27;t this mean you are not regionally isolated from us-east-1?

Doing pretty well up here in Tokyo region for now! Just can&#x27;t log into console and some other stuff.

I am down with that lets all build in US-East-1.

As they say, every cloud outage has a silver lining.* Give the computers a rest, they probably need it. Heck, maybe the Internet should just shut down in the evening so everyone can go to bed (ignoring those pesky timezone differences)* Free chaos engineering at the cloud provider region scale, except you didn&#x27;t opt in to this one and know about in advance, making it extra effective* Quickly figure out a map which of the things you use have a dependency on a single AWS region without no capability to change or re-route

This is a remarkable and unfair truth. I have had this experience with Office365...when they&#x27;re down a lot of customers don&#x27;t care because all their customers are also down.

One unexpected upside moving from a DC to AWS is when a region is down, customers are far more understanding. Instead of being upset, they often shrug it off since nothing else they needed&#x2F;wanted was up either.

A slightly less cynical view: execs have a hard filter for “things I can do something about” and “things I can’t influence at all.” The bad ones are constantly pushing problems into the second bucket, but there are legitimately gray area cases. When an exec smells the possibility that their team could have somehow avoided a problem, that’s category 1 and the hammer comes down hard.After that process comes the BS and PR step, where reality is spun into a cotton candy that makes the leader look good no matter what.

&gt; It took me so long to realise this is what&#x27;s important in enterprise. Uptime isn&#x27;t important, being able to blame someone else is what&#x27;s important.Yes.What is important is having a Contractual SLA that is defensible. Acts of God are defensible. And now major cloud infrastructure outtages are too.

“No one ever got fired for hiring IBM”

It took me so long to realise this is what&#x27;s important in enterprise. Uptime isn&#x27;t important, being able to blame someone else is what&#x27;s important.If you&#x27;re down for 5 minutes a year because one of your employees broke something, that&#x27;s your fault, and the blame passes down through the CTO.If you&#x27;re down for 5 hours a year but this affected other companies too, it&#x27;s not your faultFrom AWS to Crowdstrike - system resilience and uptime isn&#x27;t the goal. Risk mitigation isn&#x27;t the goal. Affordability isn&#x27;t the goal.When the CEO&#x27;s buddies all suffer at the same time as he does, it&#x27;s just an &quot;act of god&quot; and nothing can be done, it&#x27;s such a complex outcome that even the amazing boffins at aws&#x2F;google&#x2F;microsoft&#x2F;cloudflare&#x2F;etc can&#x27;t cope.If the CEO is down at a different time than the CEO&#x27;s buddies then it&#x27;s that Dave&#x2F;Charlie&#x2F;Bertie&#x2F;Alice can&#x27;t cope and it&#x27;s the CTO&#x27;s fault for not outsourcing it.As someone who likes to see things working, it pisses me off no end, but it&#x27;s the way of the world, and likely has been whenever the owner and CTO are separate.

And all your dependencies are co-located.

Azure outages: happens all the time, understandable, no way to prevent thisAWS outages: almost never happens, you should have been more prepared for when it does

I was once told that our company went with Azure because when you tell the boomer client that our service is down because Microsoft had an outage, they go from being mad at you, to accepting that the outage was an act of god that couldn’t be avoided.

Would think that Amazon safeguards their biggest region more, but no idea, I&#x27;ve never worked at AWS

I&#x27;ve heard this so many times and not seen it contradicted so I started saying it myself. Even my last Ops team wanted to run some things in us-east-1 to get prior warning before they broke us-west-1.But there are some people on Reddit who think we are all wrong but won&#x27;t say anything more. So... whatever.Nothing in the outage history really stands out as &quot;this is the first time we tried this and oops&quot; except for us-east-1.It&#x27;s always possible for things to succeed at a smaller scale and fail at full scale, but again none of them really stand out as that to me. Or at least, not any in the last ten years. I&#x27;m allowing that anything older than that is on the far side of substantial process changes and isn&#x27;t representative anymore.

Is us-east-1 equally unstable to the other regions? My impression was that Amazon deployed changes to us-east-1 first so it&#x27;s the most unstable region.

Choosing us-east-1 as your primary region is good, because when you&#x27;re down, everybody&#x27;s down, too. You don&#x27;t get this luxury with other US regions!

workos is down too, timing is highly correlated with AWS outage: <a href="https:&#x2F;&#x2F;status.workos.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;status.workos.com&#x2F;</a>That means Cursor is down, can&#x27;t login.

Yes <a href="https:&#x2F;&#x2F;www.vercel-status.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.vercel-status.com&#x2F;</a>

Perplexity also have outage.<a href="https:&#x2F;&#x2F;status.perplexity.ai" rel="nofollow">https:&#x2F;&#x2F;status.perplexity.ai</a>

Yes, and that&#x27;s exactly the problem. It&#x27;s like choosing a microservice architecture for resiliency and building all the services on top of the same database or message queue without underlying redundancy.

You know how people say X startup is ChatGPT wrapper? A significant chunk of AWS services are wrappers of main services (DynamoDB, EC2, S3 and etc).

Sure, but none of those other issues are ever documented by AWS as their status page is usually just one big lie.

you can&#x27;t possibly know that?surely you mean:&gt; I find it interesting that AWS services appear to be so tightly integrated that when there&#x27;s an issue THAT BECOMES VISIBLE TO ME in a region, it affects most or all services.AWS has stuff failing alllllllll the time, it&#x27;s not very surprising that many of the outages that become visible to you involve multi-system failures - lots of other ones don&#x27;t become visible!

why dont they have us east 2, 3, 4 etc. Actually have different cities.

Yeah I think there are a number of &quot;hidden&quot; dependencies on different regions, especially us-east-1. It&#x27;s an artifact of it being AWS&#x27; largest region, etc.

Sounds plausible. It&#x27;s also a &quot;fat and happy&quot; symptom not to be able to fix deep underlying issues despite an ever growing pile of cash in the company.

There&#x27;s also the issue of sometimes needing actual strong consistency. Things like auth or billing for example where you absolutely can&#x27;t tolerate eventual consistency or split-brain situations, in which case you need one region to serve as the ultimate source of truth.

I think a lot of its probably technical debt. So much internally still relies on legacy systems in US-East-1, and every time this happens I&#x27;m sure theres a discussion internally about decoupling that reliance which then turns into a massive diagram that looks like a family tree dating back a thousand years of all the things that need to be changed to stop it happening.

I find it interesting that AWS services appear to be so tightly integrated that when there&#x27;s an issue in a region, it affects most or all services. Kind of defeats the purported resiliency of cloud services.

In us-east-1? That doesn&#x27;t sound that impactful, have always heard that us-east-1&#x27;s network is a ring.Back before AWS provided transparency into AZ assignments, it was pretty common to use latency measurements to try and infer relative locality and mappings of AZs available to an account.

I remember the one where some contractor accidentally cut the trunk between AZes.

During the last us-east-1 apocalypse 14 years ago, I started awsdowntime.com - don&#x27;t make me regsiter it again and revive the page.

Friends don’t let friends use us-east-1

I can’t even login, I get the internal error treatment. This is on Amazon.de

I just ordered stuff from Amazon.de. And I highly any Amazon site can go down because of one region. Just like Netflix are rarely affected.

You might be interested in Libation [0]. I use it to de-DRM my Audible library and generate a cue sheet with chapters in for offline listening.[0] <a href="https:&#x2F;&#x2F;getlibation.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;getlibation.com&#x2F;</a>

I wanted to log into my Audible account after a long time on my phone, I couldn&#x27;t, started getting annoyed, maybe my password is not saved correctly, maybe my account was banned, ... Then checking desktop, still errors, checking my Amazon.de, no profile info... That&#x27;s when I started suspecting that it&#x27;s not me, it&#x27;s you, Amazon! Anyway, I guess, I&#x27;ll listen to my book in a couple of hours, hopefully.Btw, most parts of the amazon.de is working fine, but I can&#x27;t load profiles, and can&#x27;t login.

We use IAM Identity Center (née SSO) which is hosted in the eu-central-1 region, and I can log in just fine. Its admin pages are down, though. Ditto for IAM.Other things seem to be working fine.

I can&#x27;t log in to my AWS account, in Germany, on top of that it is not possible to order anything or change payment options from amazon.de.No landing page explaining services are down, just scary error pages. I thought account was compromised. Thanks HN for, as always, being the first to clarify what&#x27;s happening.Scary to see that in order to order from Amazon Germany, us-east1 must be up. Everything else works flawlessly but payments are a no go.

Never choose a single point of failure.Or ratherEnsure your single point of failure risk is appropriate for your business. I don&#x27;t have full resilience for my companies AS going down, but we do have limited DR capability. Same with the loss of a major city or two.I&#x27;m not 100% confident in a Thames Barrier flood situation, as I suspect some of our providers don&#x27;t have the resilience levels we do, but we&#x27;d still be able to provide some minimal capability.

seeing issues with SES in us-east-1 as well

Plenty of people last week here claiming hyperscalers are necessary while ignoring more normal hosting options.

Good thing hyperscalers provide 100% uptime.

This link works fine from Australia for me.

Can&#x27;t check out on Amazon.com.au, gives error page

Chime has completely been down for almost 12 hours.Impacting all banking series with red status error. Oddly enough, only their direct deposits are functioning without issues.<a href="https:&#x2F;&#x2F;status.chime.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;status.chime.com&#x2F;</a>

Seems the underlying issue is with DynamoDB, according to the status page, which will have a big blast radius in other services. AWS&#x27; services form a really complicated graph and there&#x27;s likely some dependency, potentially hidden, on us-east-1 in there.

The issue appears to be cascading internationally due to internal dependencies on us-east-1

Affecting Coinbase[1] as well, which is ridiculous. Can&#x27;t access the web UI at all. At their scale and importance they should be multi-region if not multi-cloud.[1] <a href="https:&#x2F;&#x2F;status.coinbase.com" rel="nofollow">https:&#x2F;&#x2F;status.coinbase.com</a>

Even railway&#x27;s status page is down (guess they use Vercel):<a href="https:&#x2F;&#x2F;railway.instatus.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;railway.instatus.com&#x2F;</a>

Just tried Perplexity and it has no answer.Damn, this is really bad.Looking forward to the postmortem.

I don&#x27;t like how they phrased it. From the Verge:“Perplexity is down right now,” Perplexity CEO Aravind Srinivas said on X. “The root cause is an AWS issue. We’re working on resolving it.”What he should have said, IMHO, is &quot;The root cause is that Perplexity fully depends on AWS.&quot;I wonder if they&#x27;re actually working on resolving that, or that they&#x27;re just waiting for AWS to come back up.

Various AI services (e.g. Perplexity) are down as well

Meanwhile my pair of 12 year old raspberry pi&#x27;s hangling my home services like DNS survive their 3rd AWS us-east-1 outage.&quot;But you can&#x27;t do webscale uptime on your own&quot;Sure. I suspect even a single pi with auto-updates on has less downtime.

and if they need West Coast, it&#x27;s us-west-2.  I consider us-west-1 to be a failed region.  They don&#x27;t get some of the new instance types, you can&#x27;t get three AZs for your VPCs, and they&#x27;re more expensive than the other US regions.

I don&#x27;t recommend to my clients they use us-east-1. It&#x27;s the oldest and most prone to outages. I usually always recommend us-east-2 (Ohio) unless they require West Coast.

your website loads for a second and then suddenly goes blank. There is one fatal errors from Framer in the console

us-east-1 was, probably still is, AWS&#x27; most massive deployment. Huge percentage of traffic goes through that region. Also, lots of services backhaul to that region, especially S3 and CloudFront. So even if your compute is in a different region (at Tower.dev we use eu-central-1 mostly), outages in us-east-1 can have some halo effect.This outage seems really to be DynamoDB related, so the blast radius in services affected is going to be big. Seems they&#x27;re still triaging.

Agreed, my company had been entirely on us-east-1 predating my joining ~12 years ago.  ~7 years ago, after multiple us-east-1 outages, I moved us to us-west-2 and it has been a lot less bumpy since then.

Anywhere other than us-east-1 in my experience is rock solid.

This is not true. There are a FEW high-level ones that have their dashboards there, but there&#x27;s a reason you log into region-specific dashboards

Trouble is one can&#x27;t fully escape us-east-1. Many services are centralized there like: S3, Organizations, Route53, Cloudfront, etc. It is THE main region, hence suffering the most outages, and more importantly, the most troubling outages.

If you&#x27;re using AWS then you are most likely using us-east-1 there is no escape. When big problems happen on us-east-1 it affect most of AWS services.

FYI: 
1. AWS IAM mutations all go through us-east-1 before being replicated to other public&#x2F;commercial regions. Read&#x2F;List operations should use local regional stacks. I expect you&#x27;ll see a concept of &quot;home region&quot; give you flexibility on the write path in the future.
2. STS has both global and regional endpoints. Make sure you&#x27;re setup to use regional endpoints in your clients <a href="https:&#x2F;&#x2F;docs.aws.amazon.com&#x2F;IAM&#x2F;latest&#x2F;UserGuide&#x2F;id_credentials_temp_region-endpoints.html" rel="nofollow">https:&#x2F;&#x2F;docs.aws.amazon.com&#x2F;IAM&#x2F;latest&#x2F;UserGuide&#x2F;id_credenti...</a>

We&#x27;re mostly deployed on eu-west-1 but still seeing weird STS and IAM failures, likely due to internal AWS dependencies.Also we use Docker Hub, NPM and a bunch of other services that are hosted by their vendors on us-east-1 so even non AWS customers often can&#x27;t avoid the blast radius of us-east-1 (though the NPM issue mostly affects devs updating&#x2F;adding dependencies, our CI builds use our internal mirror)

Is there any data on which AWS regions are most reliable? I feel like every time I hear about an AWS outage it&#x27;s in us-east-1.

Our Alexa&#x27;s stopped responding and my girl couldn&#x27;t log in to myfitness pal anymore.. Let me check HN for a major outage and here we are :^)At least when us-east is down, everything is down.

Yup&gt; We have identified the underlying issue with one of our cloud service providers.

Wonder if this is related<a href="https:&#x2F;&#x2F;www.dockerstatus.com&#x2F;pages&#x2F;533c6539221ae15e3f000031" rel="nofollow">https:&#x2F;&#x2F;www.dockerstatus.com&#x2F;pages&#x2F;533c6539221ae15e3f000031</a>

You are already logged in. If you try to access your account settings, for example, you will be disappointed...

Seems to work fine for me. I&#x27;m in Europe so maybe connecting to some deployment over here.

Can&#x27;t login to Jira&#x2F;Confluence either.

They choose their hiring-retention practices and they choose to provide global infrastructure, when is the good time to criticise them?Granted, they are not as drunk on LLM as Google and Microsoft. So, at least we can say this outage had not been vibe-coded (yet).

No we wouldn’t because there’s like a 50&#x2F;50 chance of being a H1B&#x2F;L1 at AWS. They should rethink their hiring and retention strategies.

Let&#x27;s be nice. I&#x27;m sure devs and ops are on fire right now, trying to fix the problems. Given the audience of HN, most of us could have been (have already been?) in that position.

Hah good point, pagerduty is working at least.

To everyone who was supposed to get paged but didn&#x27;t, do a postmortem, chances are your service is running via Twilio and needs migrating elsewhere.

Feel sorry for anyone with a &quot;smart&quot; coffee machine.

With how long it may last, pour some cold water on the coffee and come back to drink the supernatant in a few hr

&gt; grab a coffee and ride it outThe way things are today I&#x27;m thankful the coffee machine still works without AWS.

To everyone that got paged (like me), grab a coffee and ride it out, the week can only get better!

All the big leagues take &quot;piracy&quot; very seriously and constantly try to clamp down on it.TV rights is one of their main revenue sources, and it&#x27;s expected to always go up, so they see &quot;piracy&quot; as a fundamental threat. IMO, it&#x27;s a fundamental misunderstanding on their side, because people &quot;pirating&quot; usually don&#x27;t have a choice - either there is no option for them to pay for the content (e.g. UK&#x27;s 3pm blackout), or it&#x27;s too expensive and&#x2F;or spread out. People in the UK have to pay 3-4 different subscriptions to access all local games.The best solution, by far, is what France&#x27;s Ligue 1 just did (out of necessity though, nobody was paying them what they wanted for the rights after the previous debacles). Ligue 1+ streaming service, owned and operated by them which you can get access through a variety of different ways (regular old TV paid channel, on Amazon Prime, on DAZN, via Bein Sport), whichever suits you the best. Same acceptable price for all games.

this is how I discover that is not just Serie A doing this shenanigans. I&#x27;m not really surprised

Oh no... may be LaLiga found out pirates hosting on AWS?

Asana down
Postman workspaces don&#x27;t load
Slack affected
And the worst: heroku scheduler just refused to trigger our jobs

Looks as if they are rerouting to a different region.

Have you done anything for it to be back up? Looks like mines are still down.

Static content resolves correctly but data fetching is still not functional.

Service that runs on aws is down when aws is down. Who knew.

I had a chuckle on my way home yesterday. Standing on the train platform and seeing &quot;Next departure in: (Vercel Connection Error)&quot; on the screen. :P

Imagine using vercel, a company that literally contributes to the starvation of children and is proud of it. Also, literally just learn to use a Dockerfile and a vps, like why do these PaaS even exist, you&#x27;re paying 3x for the same AWS services.

Looks like it affected Vercel, too. <a href="https:&#x2F;&#x2F;www.vercel-status.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.vercel-status.com&#x2F;</a>My website is down :((EDIT: website is back up, hooray)

AWS CodeArtifact can act as a proxy and fetch new packages from npm when needed. A bit late for that though but sharing if you want to future proof against the yearly us-east-1 outage

Yup, was releasing something to prod and can&#x27;t even build a react app. I wonder if there is some sort of archive that isn&#x27;t affected?

Oh damn that ruins all our builds for regions I thought would be unaffected

npm and pnpm are badly affected as well. Many packages are returning 502 when fetched. Such a bad time...

Paddle (payment provider) is down as well: <a href="https:&#x2F;&#x2F;paddlestatus.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;paddlestatus.com&#x2F;</a>

No wonder, when I opened Vercel it showed a 502 error.

Presumably the root cause of the major Vercel outage too: <a href="https:&#x2F;&#x2F;www.vercel-status.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.vercel-status.com&#x2F;</a>

Well it looks like if companies and startups did their job in hiring the proper distributed systems skills more rather than hazing for the wrong skills we wouldn&#x27;t be in this outage mess.Many companies on Vercel don&#x27;t think to have a strategy to be resilient to these outages.I rarely see Google, Ably and others serious about distributed systems being down.

Maybe actually making the interviews less of a hazing ritual would help.Hell, maybe making today&#x27;s tech workplace more about getting work done instead of the series of ritualistic performances that the average tech workday has degenerated to might help too.Ergo, your conclusion doesn&#x27;t follow from your initial statements, because interviews and workplaces are both far more broken than most people, even people in the tech industry, would think.

100% wrong.Serious engineering teams that care about distributed systems and multi region deployments don&#x27;t think like this.

It seems that all the sites that ask for distributed systems in their interview and has their website down wouldn&#x27;t even pass their own interview.This is why distributed systems is an extremely important discipline.

The support apis only exist in us-east-1, iirc. It’s a “global” service like IAM, but that usually means modifications to things have to go through us-east-1 even if they let you pull the data out elsewhere.

&gt;Oct 20 12:51 AM PDT We can confirm increased error rates and latencies for multiple AWS Services in the US-EAST-1 Region. This issue may also be affecting Case Creation through the AWS Support Center or the Support API. We are actively engaged and working to both mitigate the issue and understand root cause. We will provide an update in 45 minutes, or sooner if we have additional information to share.Weird that case creation uses the same region as the case you&#x27;d like to create for.

can&#x27;t log into <a href="https:&#x2F;&#x2F;amazon.com" rel="nofollow">https:&#x2F;&#x2F;amazon.com</a> either after logging out; so many downstream issues

I can&#x27;t believe this. When status page first created their product, they used to market how they were in multiple providers so that they&#x27;d never be affected by downtime.Maybe all that got canned after the acquisition?

A lot of status pages hosted by Atlasian StatusPage are down! The irony…

Both Intercom and Twilio are affected, too.- <a href="https:&#x2F;&#x2F;status.twilio.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;status.twilio.com&#x2F;</a>
- <a href="https:&#x2F;&#x2F;www.intercomstatus.com&#x2F;us-hosting" rel="nofollow">https:&#x2F;&#x2F;www.intercomstatus.com&#x2F;us-hosting</a>I want the web ca. 2001 back, please.

Login issues with Jira Cloud here in Germany too. Just a week after going from Jira on-prem to cloud

Impossible to connect to JIRA here (France).

That&#x27;s understandably bad for anyone who depends on Ring for security but arguably a net positive for the rest of us.Amazon’s Ring to partner with Flock: <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=45614713">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=45614713</a>

Ring is affected. Why doesn’t Ring have failover to another region?

So there&#x27;s no way to get back in if you step out for food?

My apartment uses “SmartRent” for access controls and temps in our unit. It’s down…

The Ring (Doorbell) App isn&#x27;t working, nor is any the MBTA (Transit) Status pages&#x2F;apps.

Yep. It&#x27;s the auditing part that is broken. As a (dangerous) workaround use --no-audit

&quot;Thankfully current work project uses a pull-through mirror that allows us to continue working.&quot;so there is no free coffee time???? lmao

They&#x27;ve acknowledged an issue now on the status page. For me at least, it&#x27;s completely down, package installation straight up doesn&#x27;t work. Thankfully current work project uses a pull-through mirror that allows us to continue working.

Careful: NPM _says_ they&#x27;re up (<a href="https:&#x2F;&#x2F;status.npmjs.org&#x2F;" rel="nofollow">https:&#x2F;&#x2F;status.npmjs.org&#x2F;</a>) but I am seeing a lot of packages not updating and npm install taking forever or never finishing. So hold off deploying now if you&#x27;re dependent on that.

It&#x27;s never an AI&#x27;s fault since it&#x27;s up to a human to implement the AI and put in a process that prevents this stuff from happening.So blame humans even if an AI wrote some bad code.

I haven&#x27;t seen the &quot;90% of our code is AI&quot; nonsense from Amazon.

if that&#x27;s the case it&#x27;ll be buried

The fun is really gonna start if the root cause of this somehow implicates an AI as a primary cause.

The internal disruption reviews are going to be fun :)

Atlassian cloud is having problems as well.

When you know a service is down but the service says it&#x27;s up: it&#x27;s either your fault or the service is having a severe issue

Atlassian is down as well so they probably can&#x27;t access their Atlassian Statuspage admin panel to update it.

Clearly this is all some sort of mass delusion event, the Amazon Ring status says everything is working.<a href="https:&#x2F;&#x2F;status.ring.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;status.ring.com&#x2F;</a>(Useless service status pages are incredibly annoying)

Good luck to all on-callers today.It might be an interesting exercise to map how many of our services depend on us-east-1 in one way or another. One can only hope that somebody would do something with the intel, even though it&#x27;s not a feature that brings money in (at least from business perspective).

Strangely some of our services are scaling up on east-1, and there is downtick on downdetector.com so issue might be resolving.

Of course this happens when I take a day off from work lolCame here after the Internet felt oddly &quot;ill&quot; and even got issues using Medium, and sure enough <a href="https:&#x2F;&#x2F;status.medium.com" rel="nofollow">https:&#x2F;&#x2F;status.medium.com</a>

AWS&#x27;s own management console sign-in isn&#x27;t even working. This is a huge one. :(

Seems to be upsetting Slack a fair bit, messages taking an age to send and OIDC login doesn&#x27;t want to play.

Considering the history of east-1 it is fascinating that it still causes so many single point of failure incidents for large enterprises.

Also:Snapchat, Ring, Roblox, Fortnite and more go down in huge internet outage: Latest updates <a href="https:&#x2F;&#x2F;www.the-independent.com&#x2F;tech&#x2F;snapchat-roblox-duolingo-fortnite-down-not-working-b2848289.html" rel="nofollow">https:&#x2F;&#x2F;www.the-independent.com&#x2F;tech&#x2F;snapchat-roblox-duoling...</a>To see more (from the first link):
<a href="https:&#x2F;&#x2F;downdetector.com" rel="nofollow">https:&#x2F;&#x2F;downdetector.com</a>

Qwen3 on lm-studio running fine on my work Mac M3, what&#x27;s wrong with yours?

&gt; Can&#x27;t vibe code your way out of this issue...Exactly. This time, some LLM providers are also down and can&#x27;t help vibe coders on this issue.

After some thankless years preventing outages for a big tech company, I will never take an oncall position again in my life.Most miserable working years I have had. It&#x27;s wild how normalized working on weekends and evenings becomes in teams with oncall.But it&#x27;s not normal. Our users not being able to shitpost is simply not worth my weekend or evening.And outside of Google you don&#x27;t even get paid for oncall at most big tech companies! Company losing millions of dollars an hour, but somehow not willing to pay me a dime to jump in at 3AM? Looks like it&#x27;s not my problem!

Pour one out for everyone on-call right now.

Well, but tomorrow there will be CTOs asking for a contingency plan if AWS goes down, even if planning, preparing, executing and keeping it up to date as the infra evolves will cost more than the X hours of AWS outage.There are certainly organizations for which that cost is lower than the overall damage of services being down due to AWS fault, but tomorrow we will hear CTOs from smaller orgs as well.

Can confirm, pretty chill we can blame our current issues on AWS.

No really true for large systems. We are doing things like deploying mitigations to avoid scale-in (eg services not receiving traffic incorrectly autoscaling down), preparing services for the inevitable storm, managing various circuit breakers, changing service configurations to ease the flow of traffic through the system, etc. We currently have 64 engineers in our on-call room managing this. There&#x27;s plenty of work to do.

Well, some engineer somewhere made the recommendation to go with AWS, even tho it is more expensive than alternatives. That should raise some questions.

&gt; Easiest day for engineers on-call everywhereI have three words for you: cascading systems failure

Easiest day for engineers on-call everywhere except AWS staff. There’s nothing you can do except wait for AWS to come back online.Pour one out for the customer service teams of affected businesses instead

Internet, out.Very big day for an engineering team indeed. Can&#x27;t vibe code your way out of this issue...

AWS multiple services outage in us-east-1