We&#x27;re getting errors indicative of an S3 outage too.

Lol... It&#x27;s interesting how much depends on Amazon&#x27;s infrastructure at the moment.

I would assume,  I couldn&#x27;t even share a screen shot of my evidence to my team on slack!

Not sure if it&#x27;s related... but I&#x27;m having issues with Amazon Cloud-drive.

I&#x27;ve had a few non-Amazon providers tell me AWS things are not working in the last 5 minutes, no note from Amazon though.Nice.

I just check Twitter, since Amazon&#x27;s status is always a lie. My personal dashboard is still showing no problems. It&#x27;s bad enough that the main public status is always green even when there&#x27;s clearly a problem, but you&#x27;d think they could at least make the private status accurate.

Gah. It was up 3 minutes ago. Anyone have any suspicion this is another ddos episode? I saw that SO was down last night too: <a href="https:&#x2F;&#x2F;twitter.com&#x2F;StackStatus&#x2F;status&#x2F;836450836322516992" rel="nofollow">https:&#x2F;&#x2F;twitter.com&#x2F;StackStatus&#x2F;status&#x2F;836450836322516992</a>

<a href="http:&#x2F;&#x2F;downdetector.com&#x2F;status&#x2F;aws-amazon-web-services" rel="nofollow">http:&#x2F;&#x2F;downdetector.com&#x2F;status&#x2F;aws-amazon-web-services</a> looks like a reasonable alternative place to check&#x2F;report downtime.

Same here. But it is in the general status dashboard:
<a href="http:&#x2F;&#x2F;status.aws.amazon.com&#x2F;" rel="nofollow">http:&#x2F;&#x2F;status.aws.amazon.com&#x2F;</a>

There was an alert on the personal health dashboard[1] a second ago, it said S3 Operational issue in us-east-1 but when I tried to view the details it showed an error.Then I refreshed and the event disappeared altogether.[1] <a href="https:&#x2F;&#x2F;phd.aws.amazon.com&#x2F;phd&#x2F;home?region=us-east-1#&#x2F;dashboard&#x2F;open-issues" rel="nofollow">https:&#x2F;&#x2F;phd.aws.amazon.com&#x2F;phd&#x2F;home?region=us-east-1#&#x2F;dashbo...</a>

I guess sub-1% to 100% failure rate is technically an &quot;increase&quot;.

the worst thing is when your system cant handle these &quot;increased error rates&quot; as your control plane cascades failure due to something like this....The worst &quot;increased error rate&quot; problem I had was when the API was failing and my autoscale system couldnt deal and launched thousands of instances because it couldnt tell when instances were launched (lack of API access) and the instances pummelled the fuck out of all other parts of the system and we basically had to reboot the entire platform....Luckily, amazon is REALLY forgiving with respect to costs in these (and actually most) circumstance....

I&#x27;m seeing green checkmarks across the board, but they just added a notice to the top of the page:&gt; Increased Error Rates&gt; We are investigating increased error rates for Amazon S3 requests in the US-EAST-1 Region.

I guess their bizarre thinking is something along the lines of: &quot;unless we have proof that noone can access the service, we won&#x27;t change the indicator from green to yellow.Seriously: I don&#x27;t understand why you guys stay with AWS.

These service health boards are more like advertisement page then actual status of the service.

The good news is, if Amazon&#x27;s services are marked as offline, you&#x27;re allowed to use Amazon Lumberyard to control nuclear power plants.

The number of non-green marks is the number of ICBMs currently in flight towards an AWS data center.

In case anyone wants to see what mysterious the red icon looks like: <a href="https:&#x2F;&#x2F;status.aws.amazon.com&#x2F;images&#x2F;status3.gif" rel="nofollow">https:&#x2F;&#x2F;status.aws.amazon.com&#x2F;images&#x2F;status3.gif</a>At best when there are problems (not like now I guess) I will see the &quot;note&quot; green icon <a href="https:&#x2F;&#x2F;status.aws.amazon.com&#x2F;images&#x2F;status1.gif" rel="nofollow">https:&#x2F;&#x2F;status.aws.amazon.com&#x2F;images&#x2F;status1.gif</a>

I&#x27;ve heard (on the Fnord new show on the most recent CCC congress, so take it with a grain of salt and a bucket of humor) that Amazon&#x27;s TOS are more or less void when a Zombie Apocalypse breaks out.They had some convoluted but fairly specific wording in their TOS, whoever wrote must have had a lot of fun.

Then I guess it has begun, the page is now showing red.  I&#x27;d put a picture on imgur but it&#x27;s not loading.

I always joke that if one of those statuses ever went to red, it means the zombie apocalypse has begun.

Looks like his personal site isn&#x27;t loading... :)

Calling @jeffbarr<a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;user?id=jeffbarr" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;user?id=jeffbarr</a>

The truth is stranger than fiction.<a href="https:&#x2F;&#x2F;twitter.com&#x2F;awscloud&#x2F;status&#x2F;836656664635846656" rel="nofollow">https:&#x2F;&#x2F;twitter.com&#x2F;awscloud&#x2F;status&#x2F;836656664635846656</a>

Me right now: <a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=_cHa063Mwos" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=_cHa063Mwos</a>

So just as a FYI the reason that probably happened to you is that the underlying host was failing.  I am assuming they wanted to give you a window to deal with it but the host croaked before then.  I&#x27;ve been dealing w&#x2F; AWS for a long long time and I&#x27;ve never seen a maintenance event go early unless the physical hardware actually died...

That&#x27;s completely ridiculous, get some fucking RAID Amazon.I order drives off newegg directly to my DC and I&#x27;m yet to lose data with the cheapest drives available in RAID10.

That what happens when cloud provider doesn&#x27;t support live migration for VMs.

In December 2015 I received an e-mail with the following subject line from AWS, around 4 am in the morning:&quot;Amazon EC2 Instance scheduled for retirement&quot;When I checked the logs it was clear the hardware failed 30 mins before they scheduled it for retirement. EC2 and root device data was gone. The e-mail also said &quot;you may have already lost data&quot;.So I know that Amazon schedules servers for retirement after they already failed, green check doesn&#x27;t surprise me.

I really liked it. But when trying to add it to my HipChat group it failed to upload. Why? S3 outage, what an irony.

AWS internal lingo calls this the &quot;green-i&quot;

We have a slack emoji for it called greenish. It&#x27;s the classic AWS green checkmark with an info icon in the bottom. Apparently it&#x27;s NOT an outage if you don&#x27;t acknowledge it. It&#x27;s called alt-uptime.

It doesn&#x27;t appear to be global; my app in eu-west-1 appears unaffected.It&#x27;s possible that the console won&#x27;t work however as I believe that&#x27;s served from us-east-1.

So, global S3 outage for more than an hour now. Still green, still talking about &quot;US East issue&quot;. I&#x27;m amazed.

Also, <a href="https:&#x2F;&#x2F;status.pantheon.io" rel="nofollow">https:&#x2F;&#x2F;status.pantheon.io</a>

Blake Gentry gave a full accounting of Heroku&#x27;s response process here - <a href="http:&#x2F;&#x2F;www.heavybit.com&#x2F;library&#x2F;video&#x2F;every-minute-counts-coordinating-herokus-incident-response&#x2F;" rel="nofollow">http:&#x2F;&#x2F;www.heavybit.com&#x2F;library&#x2F;video&#x2F;every-minute-counts-co...</a>Amazon should take notes.

I feel for them. Imagine, 40 or 50 different engineering teams all responsible for updating their statuses. At this moment on the AWS status page I see random usage of the red, yellow, and green icons, even though all the status updates are &quot;Increased error rates.&quot; What that tells me is that there&#x27;s no unified communication protocol across the teams, or they&#x27;re not following it. And just imagine what it&#x27;s like being on the S3 team right now.I notice even Cloudflare is starting to have problems serving up pages now.

Font Awesome went out for me for a bit, but they did a great job getting back up and keeping their users in the loop.<a href="https:&#x2F;&#x2F;status.fortawesome.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;status.fortawesome.com&#x2F;</a>

It&#x27;s crazy how much better the communication (including updates and status pages) is of the companies that rely on AWS than AWS&#x27; communication itself.<a href="https:&#x2F;&#x2F;status.heroku.com&#x2F;incidents&#x2F;1059" rel="nofollow">https:&#x2F;&#x2F;status.heroku.com&#x2F;incidents&#x2F;1059</a>

Looks like they have fixed the issue with their health dashboard now.From <a href="https:&#x2F;&#x2F;status.aws.amazon.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;status.aws.amazon.com&#x2F;</a> : Update at 11:35 AM PST: We have now repaired the ability to update the service health dashboard. The service updates are below. We continue to experience high error rates with S3 in US-EAST-1, which is impacting various AWS services. We are working hard at repairing S3, believe we understand root cause, and are working on implementing what we believe will remediate the issue.

Yup, same here. It has been a few minutes already. Wanna bet the green checkmark[1] will stay green until the incident is resolved?[1] <a href="https:&#x2F;&#x2F;status.aws.amazon.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;status.aws.amazon.com&#x2F;</a>

Can&#x27;t access my website which is hosted on s3 (<a href="http:&#x2F;&#x2F;joshuajherman.com" rel="nofollow">http:&#x2F;&#x2F;joshuajherman.com</a>).

FreshDesk makes extensive use of S3 and it&#x27;s been unbearably slow to load for the past hour or so. All on S3 requests.

Uh-oh. Same here... and tried taking a screenshot of pinging s3.amazonaws.com and Slack upload hung.

They used to, but i think they are off it now.

Dropbox using this? Can&#x27;t seem to sync

It is, of course the checkmark will stay green throughout this as Amazon doesn&#x27;t care about actually letting its customers know they have a problem.

I confirm for SES being down in US-East-1 :-(

EBS is down for 30% of my servers as well

Down in US-East-1 as of 17:40 GMT. Amazon SES also down in US-East-1 as of a few minutes later.Hearing reports of EBS down as well.

Down from the outside;
The internal access (from within EC2) APIs still work.

Yup.  Some of our machines in us-east-1e dropped out of the load balancers.

in the s3 web interface requests to S3 backend end with 503 Service Unavailable

S3 and Elastic Beanstalk (S3 dependencies) ... no issues with RDS at the moment

Is cli working for anyone else?  I can&#x27;t use the console UI, but aws s3 ls and get commands seem to be working fine.

S3 Ireland (eu-west-1) seems to be doing fine at first sight.

It appears to be down.  My website runs on S3 and my monitors are going nuts!

Don&#x27;t know if share911 is still your product (its about page is down), but you could run your critical services on top of something like PagerDuty to give you the reliability you need.

Our lambda functions are also unavailable.  Lucky for us we didn&#x27;t move any of our critical functionality to lambda yet although we are planning to once we have an EC2 backup in place...

Yup it looks so. My console says I have zero buckets, my Lambdas are timing out and <a href="https:&#x2F;&#x2F;aws.amazon.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;aws.amazon.com&#x2F;</a> returns a big:&quot;500 The server encountered an error processing your request.&quot; message

Yeah, we host on S3 (US-East-1 I think) with Cloudfront for caching &#x2F; SSL. Some of our requests get through but it&#x27;s been intermittent. Lots of 504 Gateway Time-Outs when retrieving CSS, JS.

Hi Beni! My dashboard wasn&#x27;t loading (the JS assets from cloudfront.net seemed to be throwing an access-control-origin error) but it is now working again. Very slow, but it works. Fortunately nothing critical is happening right now so no worries. Love your service :)

Hi, Beni from Opbeat here. Our community site is indeed down unfortunately, but the rest should be unaffected. Where do&#x2F;did you experience issues?

circleci.com won&#x27;t even load for me.

Incredible how much stuff this affected for me. Opbeat is not loading and I can&#x27;t even deploy because CircleCI seems to depend on S3 for something and my build is &quot;Queued&quot;. This seems so dangerous...

I&#x27;m getting this using s3cmd:$ s3cmd ls 
WARNING: Retrying failed request: &#x2F; ([Errno 60] Operation timed out)
WARNING: Waiting 3 sec...
WARNING: Retrying failed request: &#x2F; ([Errno 60] Operation timed out)
WARNING: Waiting 6 sec...

&quot;I felt a great disturbance in the Force, as if millions of voices suddenly cried out in terror, and were suddenly silenced. I fear something terrible has happened.&quot;

Their status page probably can&#x27;t refresh because S3 is down.

What kills me is that their status page still shows nothing is wrong.<a href="https:&#x2F;&#x2F;status.aws.amazon.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;status.aws.amazon.com&#x2F;</a>

I get this in my aws console.Increased API Error Rates09:52 AM PST We are investigating increased error rates in the US-EAST-1 Region.Event data
Event
S3 operational issue
Status
Open
Region&#x2F;AZ
us-east-1
Start time
February 28, 2017 at 6:51:57 PM UTC+1
End time
-
Event category
Issue

Apparently they need a special&#x2F;separate status page for everyone&#x27;s personal health dashboard too. SMH.

Also seeing the intermittent event on the personal dashboard. Wonderful.

They don&#x27;t show it on the status dashboard at <a href="https:&#x2F;&#x2F;status.aws.amazon.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;status.aws.amazon.com&#x2F;</a> (at least at the time I originally posted this comment).But if you go to your personal health dashboard (<a href="https:&#x2F;&#x2F;phd.aws.amazon.com&#x2F;phd&#x2F;home#&#x2F;dashboard&#x2F;open-issues" rel="nofollow">https:&#x2F;&#x2F;phd.aws.amazon.com&#x2F;phd&#x2F;home#&#x2F;dashboard&#x2F;open-issues</a>) they report an S3 operational issue event there.Edit: Mine is reporting region us-east-1Edit 2: And now the event disappeared from my personal health dashboard too. But we are still experiencing issues. WTH.

Any specific regions?  us-west-2 seems fine to me. 
[edit] now I can&#x27;t see any of my buckets in the web interface.

Definitely experiencing non-loading for dependencies hosted on S3 at the moment...

All of our S3 assets are unavailable. Cloudfront is accessible but returning a 504 status with the message: &quot;CloudFront is currently experiencing problems with requesting objects from Amazon S3.&quot;

I&#x27;m running into timeouts trying to download elixir packages, and I&#x27;m willing to bet this is the cause

Yes, affecting elb in us-east-1 right now.  web services are down and unable to bring up the elb screen in the aws console.

I&#x27;m certainly not an expert here, but just to make you feel good (if nothing else), this is exactly what we did this morning (Melbourne time).  Woke up to a bunch of flailing Lambda funcs on us-east-1.  Luckily we&#x27;re using Apex so cross deploying them to Singapore took all of 30 seconds.  We were concerned about API Gateway since it was also sitting in us-east-1 but ended up not being an issue.  Realized that redeploying S3 and Lambda across regions can be done in practically no time, but we would have been in trouble had we needed to replicated our APIs in another region.  Going to start exploring spec&#x27;ing up our existing gateways in swagger to help with this.

To mitigate the effect of S3 going down I use cross-region replication to replicate objects to another S3 region. If S3 went down in my primary region I could update the app config to write back to the backup region and update the CDN configuration to use the backup region as an origin.I did that out of paranoia but it turns out this could happen to us. Does that sound like a sensible approach?Fortunately all my company&#x27;s stuff is in eu-west-1 which still seems to be fine.

Down for us as well. We have cloudfront in front of some of our s3 buckets and it is responding with<pre><code> CloudFront is currently experiencing problems with requesting objects from Amazon S3.

</code></pre>
Can I also say I am constantly disappointed by AWS&#x27;s status page: <a href="https:&#x2F;&#x2F;status.aws.amazon.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;status.aws.amazon.com&#x2F;</a> it seems whenever there is an issue this takes a while to update. Sometimes all you see is a green checkmark with a tiny icon saying a note about some issue. Why not make it orange or something. Surely they must have some kind of external monitor on these things that could be integrated here?edit: Since posting my comment they added a banner of&quot;Increased Error RatesWe are investigating increased error rates for Amazon S3 requests in the US-EAST-1 Region.&quot;However S3 still shows green and &quot;Service is operating normally&quot;

There&#x27;s logic behind the complicated rules. The idea is that you can&#x27;t cause the calculated availability to go down by generating lots of requests when the service is down.

Here it is, in typical SLA language... only AWS knows what it means <a href="https:&#x2F;&#x2F;aws.amazon.com&#x2F;s3&#x2F;sla&#x2F;" rel="nofollow">https:&#x2F;&#x2F;aws.amazon.com&#x2F;s3&#x2F;sla&#x2F;</a>

Many companies don&#x27;t give out refunds and you have to proactively request. (alas my employer&#x27;s value prop, in a completely different industry of course)

Does the AWS Support Center also run on AWS?

You can open tickets in support with API calls ... just sayin.

And my favourite part: &quot;To receive a Service Credit, you must submit a claim by opening a case in the AWS Support Center.&quot; -- for a company that has built itself on automation, surely you could automate some bill credits based on the SLA.

Yes. Have heard confirmation from Amazon that this outage is affecting us-east-1.

Yeah, bad morning. I started by trying to pull a Docker image. When I couldn&#x27;t, I tried to push some stuff to S3. Now of course I&#x27;m checking HN. :p

Can&#x27;t pull images from either Dockerhub or Quay right now

Yea my self hosted concourse is down. Feels quite bizarre to have our internal CI so crippled by S3... though i&#x27;m still investigating, perhaps the workers are just hung after all of the failed docker pulls.

Not sure if its related or not (I&#x27;ll just assume it is), but dockerhub is down as well. Haven&#x27;t been able to push or pull for the last 15 minutes, some other folks complaining of the same thing.

Would love to know the threshold for &quot;Increased&quot; -_-

<a href="https:&#x2F;&#x2F;twitter.com&#x2F;aws_shd&#x2F;status&#x2F;836635812020158464" rel="nofollow">https:&#x2F;&#x2F;twitter.com&#x2F;aws_shd&#x2F;status&#x2F;836635812020158464</a>&quot;Increased API Error Rates - 9:52 AM PST We are investigating increased error rates in the US-EAST-1&quot;
&quot;S3 operational issue - us-east-1&quot;

Upload failing for me from Sacramento --&gt; us-east-1

Same here, i mistakingly went to the dashboard first too. Silly me.

Their status page images are hosted on S3, so will be a while for the green checkmarks to update

Unable to log into my servers. They are still up and taking traffic, but no contact. Also unable to provision new servers at this late hour.

Same here. I can&#x27;t find any information about that though.

My EC2 Servers are also not provisioning.

We got timeouts to our bucket address from every location we tried starting at 10:37 Mountain time (GMT-7). Slack uploads started failing, imgur isn&#x27;t working, and the landing page for the AWS console is showing a 500 error in the image flipper in the middle of the page. The Amazon status page has been all green, but there is a forum post about people having problems at <a href="https:&#x2F;&#x2F;forums.aws.amazon.com&#x2F;thread.jspa?threadID=250319&amp;tstart=0" rel="nofollow">https:&#x2F;&#x2F;forums.aws.amazon.com&#x2F;thread.jspa?threadID=250319&amp;ts...</a>In the last couple of minutes that forum post has gone from not existing to 175 views and 9 posts.

Yup, same here. For a moment, I was worried that the UI showed 0 buckets. Gave me a heart attack.

I use both RS Cloud Files and Google&#x27;s Cloud Storage. Google&#x27;s is superior in nearly every way.The only con is that it is a Google product that could be deprecated at any point in time. But, with all the acquisition stuff happening over at RS, I&#x27;d be lying if I said I wasn&#x27;t worried about them killing of their cloud offering.

Rackspace&#x27;s Cloudfiles. Does support static websites.

Yes, we can. You can host static content in GCS in the same way that you would using S3, or you can Google Cloud Load Balancer to do more complex setups, such as mixing static GCS content and compute URLs on the same domain.If you&#x27;re interested:
<a href="https:&#x2F;&#x2F;cloud.google.com&#x2F;solutions&#x2F;web-serving-overview#hosting_a_static_website_with_cloud_storage" rel="nofollow">https:&#x2F;&#x2F;cloud.google.com&#x2F;solutions&#x2F;web-serving-overview#host...</a>
<a href="https:&#x2F;&#x2F;cloud.google.com&#x2F;compute&#x2F;docs&#x2F;load-balancing&#x2F;http&#x2F;using-http-lb-with-cloud-storage" rel="nofollow">https:&#x2F;&#x2F;cloud.google.com&#x2F;compute&#x2F;docs&#x2F;load-balancing&#x2F;http&#x2F;us...</a>Disclaimer: I work on Google Cloud Storage.

Google Cloud Storage comes to mind. From what I recall they can host static websites as well.

GitLab really needs to provide a way to force HTTPS for domains that have it set up. I have to manually link with `<a href="https:&#x2F;&#x2F;`" rel="nofollow">https:&#x2F;&#x2F;`</a> everywhere.

Slightly related, FYI all of the package downloads for Gitlab are timing out with a 504 error from Cloudfront (&quot;Cloudfront is having difficulty accessing S3&quot;. The registry is also showing an error.I&#x27;m trying to downgrade to an older version because our install is not working but can&#x27;t get the DEB unfortunately.

GitLab Pages [0] is pretty good for static websites IMO, you can use GitLab CI to build more complex sites as well [1].Disclosure: I work for GitLab[0]: pages.gitlab.io
[1]: <a href="https:&#x2F;&#x2F;about.gitlab.com&#x2F;2016&#x2F;12&#x2F;07&#x2F;building-a-new-gitlab-docs-site-with-nanoc-gitlab-ci-and-gitlab-pages&#x2F;" rel="nofollow">https:&#x2F;&#x2F;about.gitlab.com&#x2F;2016&#x2F;12&#x2F;07&#x2F;building-a-new-gitlab-do...</a>

As I&#x27;ve said previously: I love the Backblaze folks, but B2 is apparently hosted in a single building. So that&#x27;s not a question of if, but when.Disclosure: I work on Google Cloud.

Backblaze&#x27;s B2 is an S3 near-clone: <a href="https:&#x2F;&#x2F;www.backblaze.com&#x2F;b2&#x2F;cloud-storage.html" rel="nofollow">https:&#x2F;&#x2F;www.backblaze.com&#x2F;b2&#x2F;cloud-storage.html</a>I don&#x27;t know about reliability, but it&#x27;s a fraction of the price of S3.

Interesting that they still host the cloudron.io site on Amazon platformping cloudron.io -&gt; 54.192.7.94 -&gt; server-54-192-7-94.dfw3.r.cloudfront.net (Amazon Technologies) [1][1] <a href="http:&#x2F;&#x2F;whatismyipaddress.com&#x2F;ip&#x2F;54.192.7.94" rel="nofollow">http:&#x2F;&#x2F;whatismyipaddress.com&#x2F;ip&#x2F;54.192.7.94</a>

Use cloudron.io and install Ghost&#x2F;Wordpress&#x2F;Write your own. Just run the server wherever you want.

You also get your ssl sessions backed up on global webcaches as an added feature.

To do this you&#x27;d create a &#x27;Page Rule&#x27; that specifically tells Cloudflare to &#x27;cache everything&#x27; for a URL pattern like <a href="https:&#x2F;&#x2F;examplesite.tld&#x2F;*" rel="nofollow">https:&#x2F;&#x2F;examplesite.tld&#x2F;*</a> .You can set a cache invalidation time too.Always online is a slightly different feature I believe.

If its static site, put it anywhere and just cache it with Cloudflare. (turn on the always on feature)

IBM Bluemix offers file storage[NFS], block storage[iSCSI], and cloud storage [S3 API, Swift API].we also offer a large number of boilerplates such as Flask, ASP,net, node.jswe have been making lots of changes lately check us out!static page deploy guide: <a href="https:&#x2F;&#x2F;www.ibm.com&#x2F;blogs&#x2F;bluemix&#x2F;2014&#x2F;08&#x2F;deploying-static-web-sites&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.ibm.com&#x2F;blogs&#x2F;bluemix&#x2F;2014&#x2F;08&#x2F;deploying-static-w...</a><a href="https:&#x2F;&#x2F;console.ng.bluemix.net" rel="nofollow">https:&#x2F;&#x2F;console.ng.bluemix.net</a>

Google Cloud Storage or its Firebase hosting version.

Netlify&#x27;s status page and pre-rendering service are down due to the S3 outage.<a href="https:&#x2F;&#x2F;twitter.com&#x2F;netlifystatus&#x2F;status&#x2F;836643259053023232" rel="nofollow">https:&#x2F;&#x2F;twitter.com&#x2F;netlifystatus&#x2F;status&#x2F;836643259053023232</a>

Netlify is great for hosting static sites. Free for non-commercial sites.

Dream Objects <a href="https:&#x2F;&#x2F;www.dreamhost.com&#x2F;cloud&#x2F;storage&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.dreamhost.com&#x2F;cloud&#x2F;storage&#x2F;</a>

An open source project minio.io implements Amazon S3 APIs, and allows you to create your own in-house S3 like infra.

Any other region of AWS would also have worked around this one.

This seems like an appropriate time as any... Anyone want to list some competitors to S3? Bonus if it also provides a way to host a static website.

The AWS status page is still showing all green but how has a header saying they are investigating increased error rates. <a href="https:&#x2F;&#x2F;status.aws.amazon.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;status.aws.amazon.com&#x2F;</a>

Openstack Swift - easy to get setup and reliable. Takes a little expertise but was well worth the investment in learning it!

This is one of the times that I am glad to be running my own distributed object storage. I&#x27;m sure it&#x27;s not as robust as Amazon, but......

Me too. This couldn&#x27;t have come at a worse time. Just launched a new site.

It is, they are downloading slugs from S3.

One of my heroku apps is down, and I cant&#x27; log into the heroku dashboard to check it out. I&#x27;m guessing this is related.

Yes it&#x27;s down for me. I can&#x27;t access files stored on S3. Also, the service I run is hung trying to store files on S3.

Dropbox is down as well. This is going to be gud.

they&#x27;re down to 3 ninesedit: for the year, it only takes 52.57 minutes

I didn&#x27;t get that at all from the OP. His comment I&#x27;m fairly certain is of the &quot;why the heck are we centralizing the web to 2 or 3 infrastructure companies&quot; for what amounts to a minor amount of convenience.We&#x27;ve seen this story play out in other industries and it never works out well for average people. It&#x27;s been astounding for me to watch the pace of this centralizing and who is helping it along.The tldr; point is that a single service provider should not have the amount of control Amazon does over the Internet. At least that&#x27;s my take.I know my opinion on this wildly differs from the HN crowd and SV &quot;decision makers&quot; these days - what is so curious to me is that this is a complete 180 from that same demographic even 10 years ago.

&quot;To big [to allow] to fail&quot; is what that term means.This would extend to a service like amazon actually, where survival of the service would be an extraordinary effort in case this problem lasted for a long time.The way you imagined it, as 100% uptime, is incorrect.

The fact that we even have &quot;key&quot; services is absurd.

Its unreal watching key web services fall like dominoes. Its too bad the concept of &quot;too big to fail&quot; applies only to large banks and countries.

Thanks for posting this. I&#x27;ve passed this information through my network.Slack image uploads are hanging.

Still down for us. S3 seems to be the only thing affected - our mobile apps work fine (EC2 and RDS backend)

Everything is green. There are no tanks in Baghdad center.

Yes.&quot;We&#x27;ve identified the issue as high error rates with S3 in US-EAST-1, which is also impacting applications and services dependent on S3. We are actively working on remediating the issue.&quot;

Have they even acknowledged the mortem..?

Agreed.  AWS&#x27; postmortems are fascinating.

Best thing about incidents like these: post-mortems for systems of this scale are absolutely fascinating. Hopefully they publish one.

Yah getting the same error in multiple regions as of 1:12 EST

Apologizes for the &quot;me too&quot; post:It appears to be impacting gotomeeting, I get this error when trying to start a 12pm meeting here:CloudFront is currently experiencing problems with requesting objects from Amazon S3.Edit: ironically, my missed 12pm meeting was an Azure training session.

Yes serious API problems started about 15 minutes ago.  Around noon central.

Seeing it in eu-west-1 as well. Even the dashboard won&#x27;t load. Shame on AWS for still reporting this as up; what use is a Personal Health Dashboard if it&#x27;s to AWS&#x27;s advantage not to report issues?

Huh, I&#x27;m not seeing it on my us-west-2 services.  Interesting.

The dashboard doesn&#x27;t load, nor does content using the generic S3 url [1], but we&#x27;re in us-west-2 and it works fine if you use the region specific URL [2]. In practice this means our site on S3&#x2F;Cloudfront is unaffected.[1]: <a href="https:&#x2F;&#x2F;s3.amazonaws.com&#x2F;restocks.io&#x2F;robots.txt" rel="nofollow">https:&#x2F;&#x2F;s3.amazonaws.com&#x2F;restocks.io&#x2F;robots.txt</a>[2]: <a href="https:&#x2F;&#x2F;s3-us-west-2.amazonaws.com&#x2F;restocks.io&#x2F;robots.txt" rel="nofollow">https:&#x2F;&#x2F;s3-us-west-2.amazonaws.com&#x2F;restocks.io&#x2F;robots.txt</a>

IIRC the console for S3 is global and not region specific even though buckets are.

Same here, and it&#x27;s 100% consistent, not &#x27;increased error rates&#x27; but actually just fully down. I&#x27;d just stop working but I have a demo this afternoon... the downsides of serverless&#x2F;cloud architectures, I guess.

Our services in us-west-2 have been up the whole time.I think the problem is globally accessible APIs are impacted. As others have noted, if you can use region&#x2F;AZ-specific hostnames to connect, you can get though to S3.CloudFront is faithfully serving up our existing files even from buckets in US-East.

I&#x27;m getting the same outage in us-west-2 right now.

Reminds me of an even older joke (from 80&#x27;s or 90&#x27;s):Q: Why computers don&#x27;t crash at the same time?A: Because network connections are not fast enough.(I think we are starting to get there)

Reminds me of an old joke: Why do we host on AWS? Because if it goes down then our customers are so busy worried about themselves being down that they don&#x27;t even notice that we&#x27;re down!

Probably valid, though in this case while us-west-1 is still serving my static websites, I can&#x27;t push at all.

us-east-2 is brand new and us-east-1 is the oldest region. Any time there is an issue, it is almost always us-east-1. If possible, I would migrate out of us-east-1.

Is us-east-2 (Ohio) any better (minus this aws-wide S3 issue)?

If you have a piece of unvarnished wood handy...

Amen. We setup our company cloud 2 years ago in US-West-2 and have never looked back. No outage to date.

Really? Even Australia? Can you provide evidence of this so I know for any clients that call me today? :)EDIT: Found my answer. &quot;Just to stress: this is one S3 region that has become inaccessible, yet web apps are tripping up and vanishing as their backend evaporates away.&quot; -- <a href="https:&#x2F;&#x2F;www.theregister.co.uk&#x2F;2017&#x2F;02&#x2F;28&#x2F;aws_is_awol_as_s3_goes_haywire&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.theregister.co.uk&#x2F;2017&#x2F;02&#x2F;28&#x2F;aws_is_awol_as_s3_g...</a>

I believe the reports here are misleading: if you try to access your other regions through the default s3.amazonaws.com it apparently routes through us-east first (and fails), but you&#x27;re &quot;supposed to&quot; always point directly at your chosen region.Disclosure: I work on Google Cloud (and didn&#x27;t test this, but some other comment makes that clear).

It shouldnt be technically possible to lose S3 on every region, how did amazon screw this up so bad?

Two different vendors if you can afford it. It&#x27;s a bit of a hassle though.

But now you&#x27;re talking about added effort.  Multi-AZ on AWS is easy and fairly automatic, multi-region (and multi-provider) not so much.  It&#x27;s easy to say things like this, but people who can do ops are not cheap and plentiful.

My advice is: don&#x27;t keep your eggs in one basket. AZs a localised redundancy, but as Cloud is cheap and plentiful, you should be using two or more regions, at least, to house your solution (if it&#x27;s important to you.)EDIT: less arrogant. I need a coffee.

A piece of hard-earned advice: us-east-1 is the worst place to set up AWS services. You&#x27;re signing up for the oldest hardware and the most frequent outages.For legacy customers, it&#x27;s hard to move regions, but in general, if you have the chance to choose a region other than us-east-1, do that. I had the chance to transition to us-west-2 about 18 months ago and in that time, there have been at least three us-east-1 outages that haven&#x27;t affected me, counting today&#x27;s S3 outage.EDIT: ha, joke&#x27;s on me. I&#x27;m starting to see S3 failures as they affect our CDN. Lovely :&#x2F;

Getting Issues with Citrix Sharefile api (which I&#x27;ve suspected to run in S3). Seems to only be impacting writes in preliminary assessment.

I like how this post says &quot;if you look at the AWS Status Page, this what you see&quot;. but you can&#x27;t see the image. because S3 is down.

The AWS Status page will lie to you: <a href="https:&#x2F;&#x2F;medium.com&#x2F;@ev.dev.dev&#x2F;the-aws-status-page-will-lie-to-you-4c24a68d8e0a#.1vr972hcx" rel="nofollow">https:&#x2F;&#x2F;medium.com&#x2F;@ev.dev.dev&#x2F;the-aws-status-page-will-lie-...</a>

While that may be true, that&#x27;s not the reason you&#x27;re seeing green. You should have been seeing a broken image or a status page not finishing loading if that was an issue.

FYI, in seriousness you can see the fabled red status icon here:<a href="https:&#x2F;&#x2F;status.aws.amazon.com&#x2F;images&#x2F;status3.gif" rel="nofollow">https:&#x2F;&#x2F;status.aws.amazon.com&#x2F;images&#x2F;status3.gif</a>It does exist, apparently.

There are some real gems @Pinboard too: &quot;Green checkmark = no lava in data center. Green checkmark with information icon = data center filling with lava <a href="https:&#x2F;&#x2F;status.aws.amazon.com&quot;" rel="nofollow">https:&#x2F;&#x2F;status.aws.amazon.com&quot;</a>

Apparently this is not a joke:“The dashboard not changing color is related to S3 issue. See the banner at the top of the dashboard for updates.”<a href="https:&#x2F;&#x2F;twitter.com&#x2F;awscloud&#x2F;status&#x2F;836656664635846656" rel="nofollow">https:&#x2F;&#x2F;twitter.com&#x2F;awscloud&#x2F;status&#x2F;836656664635846656</a>

I thought this was a joke, but apparently not too far off:&quot;The dashboard not changing color is related to S3 issue. See the banner at the top of the dashboard for updates.&quot;<a href="https:&#x2F;&#x2F;twitter.com&#x2F;awscloud&#x2F;status&#x2F;836656664635846656" rel="nofollow">https:&#x2F;&#x2F;twitter.com&#x2F;awscloud&#x2F;status&#x2F;836656664635846656</a>

@mikecb on Twitter explained it well. &quot;The red icon is stored in S3 US East.&quot;

when you start investigating an outage, that is exactly when you should change your checkmark to yellow if not red. if you&#x27;re as big as AWS there should not be any more than a minute or two between when your service goes down and when you actually update your status page to show that.

After having seen multiple AWS outages&#x2F;service disruptions, with nothing other than a green checkmark ever showing, I am now very confident that the checkmarks are hardcoded and there is no logic behind them.

I don&#x27;t think they intentionally kept the checkmarks there. They probably just didn&#x27;t update it as quickly as developers made a post on Hacker News (not surprising, they were probably investigating).

the non-green icon is probably hosted on s3 (i&#x27;m not trying to be funny)

I&#x27;m curious about what happened to products that were no longer fit for sale, but still fit for use. Do you recall?

&gt; Amazon, everything is on fire. you are not fooling anyoneFun story, when I was an intern at Amazon there was actually a warehouse fire. The result was a lot of manual database entry updating as products were determined to be destroyed or still fit for sale.

And system monitoring which isn&#x27;t dependent on itself. Kind of a &quot;duh&quot; kind of thing...

<a href="http:&#x2F;&#x2F;outage.report&#x2F;" rel="nofollow">http:&#x2F;&#x2F;outage.report&#x2F;</a> does this pretty much, except for the &quot;verified customers&quot; part.

What we need are status pages that are driven by votes from verified customers, which could also serve to inform the provider about issues.This would address issues that are only visible from the outside.

If this isn&#x27;t good evidence that amazon downright lies on their status page and that no green checkmark should ever be considered trustworthy, I don&#x27;t know what is.

&gt; edit: in the future, please subscribe to @MyFootballNow for timely AWS service status updates <a href="https:&#x2F;&#x2F;pbs.twimg.com&#x2F;media&#x2F;C5xdm9_WMAAY7y_.jpg:large" rel="nofollow">https:&#x2F;&#x2F;pbs.twimg.com&#x2F;media&#x2F;C5xdm9_WMAAY7y_.jpg:large</a>So this is what centralization looks like.

it would appear that you are correct&quot;The dashboard not changing color is related to S3 issue. See the banner at the top of the dashboard for updates.&quot; <a href="https:&#x2F;&#x2F;twitter.com&#x2F;awscloud&#x2F;status&#x2F;836656664635846656" rel="nofollow">https:&#x2F;&#x2F;twitter.com&#x2F;awscloud&#x2F;status&#x2F;836656664635846656</a>

I hear that their process for updating the status page involves S3.

So... <a href="https:&#x2F;&#x2F;twitter.com&#x2F;awscloud&#x2F;status&#x2F;836656664635846656" rel="nofollow">https:&#x2F;&#x2F;twitter.com&#x2F;awscloud&#x2F;status&#x2F;836656664635846656</a>&quot;The dashboard not changing color is related to S3 issue. See the banner at the top of the dashboard for updates.&quot;

Maybe you could encourage your colleagues to host the status page outside of AWS?

Could you go more in depth? What does S3 have anything to do with it?

For some reason, reading about &quot;believe we understand root cause&quot; made me think of: &quot;A distributed system is one in which the failure of a computer you didn&#x27;t even know existed can render your own computer unusable.&quot;

(Disclaimer: I work for AWS.)The dashboard is not changing color due to the S3 issue. We&#x27;re updating the banner in place of that.Edit: Update at 11:35 AM PST: We have now repaired the ability to update the service health dashboard. The service updates are below. We continue to experience high error rates with S3 in US-EAST-1, which is impacting various AWS services. We are working hard at repairing S3, believe we understand root cause, and are working on implementing what we believe will remediate the issue.<a href="http:&#x2F;&#x2F;status.aws.amazon.com&#x2F;" rel="nofollow">http:&#x2F;&#x2F;status.aws.amazon.com&#x2F;</a>

I love it, like that fixes the problem!
..now fix the REAL problem

&quot;Update at 11:35 AM PST: We have now repaired the ability to update the service health dashboard.&quot;Yep.

Why is the status board hosted on AWS? Most providers host such pages on a 3rd party, specifically for this reason; correlated failure.

what&#x27;s truly incredible is that S3 has been offline for h̶a̶l̶f̶ ̶a̶n̶ ̶h̶o̶u̶r̶ two hours now and Amazon still has the audacity to put five shiny green checkmarks next to S3 on their service page.they just now put up a box at the top saying &quot;We are investigating increased error rates for Amazon S3 requests in the US-EAST-1 Region.&quot;increased error rates? really?Amazon, everything is on fire. you are not fooling anyoneedit: in the future, please subscribe to @MyFootballNow for timely AWS service status updates <a href="https:&#x2F;&#x2F;pbs.twimg.com&#x2F;media&#x2F;C5xdm9_WMAAY7y_.jpg:large" rel="nofollow">https:&#x2F;&#x2F;pbs.twimg.com&#x2F;media&#x2F;C5xdm9_WMAAY7y_.jpg:large</a>

yeah, looks like Travis CI is down, too: <a href="https:&#x2F;&#x2F;www.traviscistatus.com" rel="nofollow">https:&#x2F;&#x2F;www.traviscistatus.com</a>

Corporate language is entertaining while we all pull out our hair.&quot;We are investigating increased error rates for Amazon S3&quot; translates to &quot;We are trying to figure out why our mission critical system for half the internet is completely down for most (including some of our biggest) customers.&quot;

I&#x27;m getting{
 &quot;errorCode&quot; : &quot;InternalError&quot;
}When I attempt to use the AWS Console to view s3

Ask HN: Is S3 down?