It has to deal with the number of open connections to postgresql, not disk IO. You don't want to have 4,000 connections open at once (unless you were using a connection balancer, but that would only work in transaction mode).

BTW, here's a counter-intuitive solution if using pgbouncer is not possible. Simply drop and reestablish connections on every request.In theory this is horrible, since PG connections are so expensive. In practice the cost of establishing a connection is negligible for a Rails app.I do suspect this will make performance "fall-off-the cliff" as you get close to capacity.

Postgres has a limitation on number of open connections.  This is because 1 connection = 1 process.  MySQL uses threads, which scales better but has other downsides.  The thread-based approach is also possible with Postgres using 3rd party connection pooling apps, e.g. pgbouncer.

Each connection forks a new process on the server, not a thread.

Data Dep't here. Postgres scales great on Heroku, AWS, and in general. We've got users doing many thousands of query per second, and terabytes of data. Not a problem.The issue with the number of connections is that each connection creates a process on the server. We cap the connections at 500, because at that point you start to see problems with O(n^2) data structures in the Postgres internals that start to make all kinds of mischief. This has been improved over the last few releases, but in general, it's still a good idea to try and keep the number of concurrent connections down.*EDIT: thanks. not a thread. :)

I am only going to suggest a small edit -&#62; s/Postgres can’t/Heroku's Postgres can't/PG can scale up pretty well on a single box, but scaling PG on AWS can be problematic due to the disk io issue, so I suspect they just don't do it. I'd love to be corrected :)

People invested too much time/money/energy into Heroku and get defensive when a problem needs discussion? Just guessing.

I think this can happen when its been erroneously flagged as having been voting-ringed.

Hmm seems to be back on the homepage again, not seen that before

This has a weird habit of happening - or getting noticed at least - when the discussion concerns YC companies.

Shame that this seems to have been flagged off the homepage before a reasonable discussion can ensue

Given how bold the claims are, I'd say it'd be best if Heroku reacted quickly, indeed. Cause right now, it seems that the author is either not understanding how Heroku works or ignoring important specifics/nuances inherent to their "secret-sauce" algorithms. Whether the analysis done in the article is sound or the author is actually pushing an agenda remains to be discussed and the sooner the better.I'm also quite wary of the incentive a 20K monthly bill would give you to try and shake Heroku down for a rebate. By the way, the figure in itself seems very high, but out of context it's impossible for me, the reader, to evaluate if that's actually good money or not. Maybe other solutions (handling everything yourself) would actually be WAY more costly, maybe Heroku actually provides a service that is well-worth the money or maybe the author is right and it's actually swindling on Heroku's part, no way to know.

But then again, those two links don't address the core of the problem:Heroku is used by tons of people around the world. Some of them are paying good money for the service. Given the amount of scrutiny under which they operate, what is the incentive for them to turn an algorithm into a less effective one and still charge the same amount of money in a growing "cloud economy" where companies providing the same kind of service are a dime a dozen (AWS, Linode, Engine Yard, etc)?How does that benefit their business if "calling their BS" is as easy as firing Apache Benchmark, collecting results, drawing a few charts and flat out "prove" that they're lying about the service they provide??I mean, I doubt Heroku is that stupid, they know how their audience doesn't give them much room for mistakes. So as nice as the story sounds on paper, I'd really like another take on all this, either from other users of Heroku, independent dev ops, researchers, routing algorithms specialists or even Heroku themselves before we all too hastily jump to sensationalist conclusions.

This is not a new revelation. I got them to admit to it 2 years ago.<a href="http://tiwatson.com/blog/2011-2-17-heroku-no-longer-using-a-global-request-queue" rel="nofollow">http://tiwatson.com/blog/2011-2-17-heroku-no-longer-using-a-...</a>and specifically:<a href="https://groups.google.com/forum/?fromgroups#!msg/heroku/8eOosLC5nrw/Xy2j7GapebIJ" rel="nofollow">https://groups.google.com/forum/?fromgroups#!msg/heroku/8eOo...</a>

Someone from Heroku really needs to weigh in on this.

Thanks for the blog post, by the way. When we were struggling with our own Heroku scaling issues last year (we eventually moved to AWS), I came across it and it was good vindication that somebody else was facing the same issue.

Even their own docs were wrong on this for a long time. It bit me in the ass back in 2011 and I got them to clarify and update the documentation just a little.<a href="http://tiwatson.com/blog/2011-2-17-heroku-no-longer-using-a-global-request-queue" rel="nofollow">http://tiwatson.com/blog/2011-2-17-heroku-no-longer-using-a-...</a>

The requests will not be served at the same time, that's the whole point. If a request is routed to a busy dyno, you will have to wait that the previous job finish before being able to start yours.

Yes it does sound as horrific.  It should never route more than N requests to a dyno, where N is the number of requests it can handle simultaneously.  It doesn't matter if N is 1, 2, or 10.  And it should always send new requests to the emptiest dyno.

&#62; But really, throwing in the towel at intelligent routing and replacing it with "random routing" is horrific, if true.The thing is, their old "intelligent routing" was really just "we will only route one request at a time to a dyno." In other words, what changed is that they now allow dynos to serve multiple requests at a time. When you put it that way, it doesn't sound as horrific, does it?

So heroku doesn't want these $20,000/mo accounts, just technically understaffed startups paying $1000/mo?I think heroku does want to be a long-term technology choice.

&#62; That's one reason customers are throwing money at you to solve it, Heroku.People are throwing money at Heroku because it's really easy to use, not because it's the best long-term technology choice. Seriously - what percentage of Heroku paying users do you think actually read up on the finest technical details like routing algorithms before they put in their credit card? Heroku knows. They know you can't even build a highly-available service on top of it, since it's singly-homed, and they're still making tons of money.

I'm sure it's because the idea of heroku is so great.

Because the premise is a good one, abstracting away some of the lower level details so people can just worry about building their apps. It has always been a tough abstraction to pull off though. It is possible the answer is simply that rap genius has outgrown heroku, in the same way as twitter outgrowing rails it doesn't make it a bad tool for everyone.

Why do you 'want to love' Heroku? Because their marketing speak is so great?

This should be more prominent. I want to love Heroku, and am sure that I could.But really, throwing in the towel at intelligent routing and replacing it with "random routing" is horrific, if true.It's arguable that the routing mesh and scaling dynamics of Heroku are a large part, if not -the- defining reason for someone to choose Heroku over AWS directly.Is it a "hard" problem? I'm absolutely sure it is. That's one reason customers are throwing money at you to solve it, Heroku.

This is something that I have been struggling with the past long while. Very troublesome when a dyno cycles itself (like they always will at least every 24 hours), as the routing layer continues to send it requests, resulting in router level "Request Timeouts" if it takes too long to restart.Especially difficult to diagnose when the queue and wait time in your logs are 0. What is the point of these in the logs if it never waits or queues?

We moved our Twilio app off Heroku for the same reasons. Extensive optimizations and we would still get timeouts on Twilio callbacks.The routing dynamics should be explained better in Heroku's documentation. From an engineering perspective, they're a very important piece of information to understand.We're with <a href="https://bluebox.net" rel="nofollow">https://bluebox.net</a> now and are very happy.

We ran into this exact same problem at Impact Dialing. When we hit scale, we optimized the crap out of our app; our New Relic stats looked insanely fast, but Twilio logs told us that we were taking over 15 seconds to respond to many of their callbacks. After spending a few weeks working with Heroku support (and paying for a dedicated support engineer), we moved to raw AWS and our performance problems disappeared. I want to love Heroku, but it doesn't scale for Rails apps.

But you never want to have a queue on any of your dynos! A queued request means that a user is waiting with no response. If your goal is to have 0 (or less than epsilon) requests queued, it takes far fewer dynos if the requests are routed intelligentlyIf you have 10 dynos and 1000 simultaneous requests, the difference between naive and intelligent might well be reduced, but that's also a scenario in which your end user response times would be horrendously slow and so you'd need more dynos either way

I don't think the wording's great. It's not throughput that's important, it's throughput at an acceptable latency.You don't need 50x as many dynos to get the same throughput, you need 50x as many dynos to get the same latency characteristics at that throughput.

i don't think some details of the argument hold. it alleges that you need more dynos to get the same throughput. but that's not true once you have sufficient demand to keep a queue of about sqrt(n) (i think - someone who knows more theory than me can correct me) in size on the dyno (where you have n dynos). because at that point all dynos will be running continuously, and the throughput will be the same with either routing.the average latency will be higher, though (and the spread in latency larger).

I have a Capistrano config that I can slab in my Rails projects. I can then do a:cap deploy:setupcap deploy:coldcap deployAnd now my app is running on my server, I then add routing and I am good to go.It is less fancy than Heroku if you want to play with some new technology, you need to install it, and get it configured, and get it to run properly.

Personally - I prefer Linode to Heroku, sure there is more of my time consumed with sys admin, but I like having full control over my platform &#38; setup, rather than having it virtually dictated to me. I'm always open to change but this strategy has served me very well for almost 3 years now.

I can chime in here that I have had similar experiences at another large scale place :).  Some requests would take a second or more to complete with the vast majority finishing in under 100MS.  A solution was put in place that added about 5 MS to the average request, but also crushed the long tail(it just doesn't even exist anymore) and everything is hugely more stable and responsive.

I seem to recall Google mentioning on some blog several years ago that high variance in response latency degrades user experience much more than slightly higher average request times.  I can't find the link though; if anyone has it, I'd be grateful.

Correct, this matches my observations as well. I'd trade an increase in mean latency for a decrease in worst-case latency anytime. It makes it so much easier to reason about how many resources are needed for a given workload when your latency is bounded.

That's probably true, but the value that Heroku is selling (and they charge a lot for it!) is that you _don't_ need to deal with this - that they will balance that out for you.

&#62; Narrowing your latency band --even if it makes the average case worse-- makes so many systems problems better (top of the list: load balancing) it's not even funny.Yeah, it's a lot more practical than implementing QoS, isn't it?

I do perf work at Facebook, and over time I've become more and more convinced that the most crucial metric is the width of the latency histogram. Narrowing your latency band --even if it makes the average case worse-- makes so many systems problems better (top of the list: load balancing) it's not even funny.

Wouldn't the bad tails of random routing be an unpredictably long length of time since long running requests times are unpredictable?Even if you work on narrowing the fat tails, shouldn't you still need to be upfront and clear about how adding a new dyno only gives you an increased chance of better request handling times as you scale?

I think the problem is that any servers which can handle concurrent requests now need to decide how many requests they can handle. Since most application servers seem to have concurrency values of "1, ever" or "I dunno, lots" this is a hard problem.Your solution would likely work if you had some higher level (application level? not real up on Heroku) at which you could specify a push vs. pull mechanism for request routing.

Re the distribution, absolutely. That "FIFTY TIMES" is totally due to the width of the distribution. Although, you know, even if their app was written such that every single request took exactly 100ms of dyno time, this random routing would create the problem all over again, to some degree.As for the intelligent routing, could you explain the problem? The goal isn't to predict which request will take a long time, the goal is to not give more work to dynos that already have work. Remember that in the "intelligent" model it's okay to have requests spend a little time in the global queue, a few ms mean across all requests, even when there are free dynos.Isn't it as simple as just having the dynos pull jobs from the queue? The dynos waste a little time idle-spinning until the central queue hands them their next job, but that tax would be pretty small, right? Factor of two, tops? (Supposing that the time for the dyno-initiated give-me-work request is equal to the mean handling time of a request.) And if your central queue can only handle distributing to say 100 dynos, I can think of relatively simple workarounds that add another 10ms of lag every factor-of-100 growth, which would be a hell of a lot better than this naive routing.What am I missing?

Simulation author here with some additional analysis using a faster distribution of request times. If you use a distribution with median 50 ms, 90th percentile 225 ms, and 99.9th percentile 898 ms, then you need 30 intelligent dynos to handle 9000 requests/minute without queueing. In the same scenario with 30 naive dynos, 44% of requests get queued.Animations and results are in the explanation at <a href="http://rapgenius.com/1502046" rel="nofollow">http://rapgenius.com/1502046</a>

Wow, it's annoying to get modded down for saying something correct. Don't believe me? Run a golang program with strace -f sometime.You could write a pthreads-compliant threading library without using threads at all, just epoll.

&#62; The solution here is to figure out why your 99th is 3 seconds. Once you solve that, randomized routing won't hurt you anymore. You hit this exact same problem in a non-preemptive multi-tasking system (like gevent or golang).The Golang runtime uses non-blocking I/O to get around this problem.

That isn't the only problem with random routing - the problems aren't as pronounced with uniform response speeds, but you still get a significant difference in net effective queue time, especially if you're operating close to your throughput limit.

Yes, it is very hard to do it at scale, but so what?
I mean, isn't the whole premise of their company to do intelligent things at scale so you don't have to?It's not an insurmountable problem by any measure, and it's definitely worth it.

The solution here is to figure out why your 99th is 3 seconds.I'm not sure this applies to the OP. His in-app measurements were showing all requests being handled very fast by the app itself; the variability in total response time was entirely due to the random routing.

So the issue here is two-fold:
- It's very hard to do 'intelligent routing' at scale.
- Random routing plays poorly with request times with a really bad tail (median is 50ms, 99th is 3 seconds)The solution here is to figure out why your 99th is 3 seconds. Once you solve that, randomized routing won't hurt you anymore. You hit this exact same problem in a non-preemptive multi-tasking system (like gevent or golang).

That's essentially round-robin, but it still requires syncing the data of which dyno is next, which is probably what they're trying to avoid.

I'm not familiar with minimum-of-two-random routing, but it does seem like assigning request to dynos in sequence would perform much better than assigning randomly (ie in a scenario with n dyno capacity, request 1 =&#62; dyno 1, request 2 =&#62; dyno 2, ... request n =&#62; dyno n, request n+1 =&#62; dyno 1, ..., repeat)That'd be probably significantly better than the case of (request i =&#62; dyno picked out of hat) for all i

If Heroku had the data needed to do minimum-of-two random routing, they'd have the data needed to do intelligent routing.Not strictly true; imagine that they can query the load state of a dyno, but at some non-zero cost. (For example, that it requires contacting the dyno, because the load-balancer itself is distributed and doesn't have a global view.)Then, contacting 2, and picking the better of the 2, remains a possible win compared to contacting more/all.See for example the 'hedged request' strategy, referenced in a sibling thread by nostradaemons from a Jeff Dean Google paper, where 2 redundant requests are issued and the slower-to-respond is discarded (or even actively cancelled, in the 'tiered request' variant).

If Heroku had the data needed to do minimum-of-two random routing, they'd have the data needed to do intelligent routing.  The problem is not the algorithm itself: "decrement and reheap" isn't going to be a performance bottleneck.  The problem is tracking the number of requests queued on the dyno.

Randomized routing isn't all bad. In fact, if Heroku were to switch from purely random routing to minimum-of-two random routing, they'd perform asymptotically better [1].[1]: <a href="http://www.eecs.harvard.edu/~michaelm/postscripts/mythesis.pdf" rel="nofollow">http://www.eecs.harvard.edu/~michaelm/postscripts/mythesis.p...</a>

not if you use elastic load balancer... it's incredibly easy

Setting up SSL on Elastic Beanstalk was very easy for us. The documentation explained the entire process. It is easier if you get a wildcard SSL cert, so then you can use the same SSL cert for your various deployments under the same domain.

SSL pain can be a major pain to set up. Is the process of setting it up remotely easy compared to Heroku?

I found Elastic Beanstalk had extremely serious latency and performance problems for PHP5, that didn't occur when setting up manual EC2/Load balancer. I intended to investigate it, but never had time.

I spent a lot of time trying to come up with a stable installation of EB for a Rails 3.2 app using RDS and just couldn't get it to a state where I'd ever deploy it as production. Here's where I decided to pull the plug:<a href="https://forums.aws.amazon.com/thread.jspa?messageID=410445&#38;#410445" rel="nofollow">https://forums.aws.amazon.com/thread.jspa?messageID=410445&#...</a>Quite a few people still report a myriad of issues with Rails applications. I really want EB to work well with Rails, but just didn't have confidence in it.

Given that ElasticBeanstalk has support for rails now, does Heroku still have any advantage over AWS for a new startup?

This explains why some of my benchmarking tools gave very different and sometimes weird results when figuring out how many dynos our application needed.As this blogpost also states Heroku really need to keep their documentation up to date. I sometimes stumble across something old referring to an old stack, or something contradicting.

rails server -p $PORT
rake jobs:work
Webrick and DJNo procfile? No Unicorn or Puma? No worker process or threads defined

Not only that, East Coast vs. West Coast at that...

We've had Battle Rap, Gangsta Rap, you name it Rap.Maybe this is the start of Off the Rails Rap.

Rap Genius is employing a classic rap-mogul strategy: start a beef

It doesn't appear that running on Heroku is free for them in terms of time.

Rap Genius is limited more by time than by money if anything. It would make more sense to throw money at the problem instead of people.

Wow. I suspect Rap Genius has the dollars now where it's totally feasible for them to go beyond Heroku, but it still might not be the best use of their time. But if they have to do it, they have to do it.OTOH, having a customer have a serious problem like this AND still say "we love your product! We want to remain on your platform", just asking you to fix something, is a pretty ringing endorsement. If you had a marginal product with a problem this severe, people would just silently leave.

It would be hard to say without a lot of assigned resources (money) and proper testing.

Is there a reason for using both HAproxy and Nginx?

Engine Yard is more like opinionated configuration management. It allocates and configures EC2 instances that you can log into like normal. The software stack is HAProxy, nginx, unicorn, etc, and customizable through the web interface and/or chef.

How does this compare to EngineYard/AppFog/any other Heroku competitors?

They are. It's in their DNA to criticize (seamlesswebsucks.com)... But they're also correct. So many reasons to hate them ;)

I think that goes with the whole rap deal. They should have challenged them to a dance-off.

Interested in how you achieved this. Did you change server setup significantly from the default EY stack?

And how much more engineering time do you waste on it now?

I've been around a few companies migrating from EY/Heroku -&#62; AWS and the cost effectiveness is always astonishing.  In addition you gain full control over your architecture, which is a major plus.

we just switched 1/3rd of our infrastructure off our existing host (engineyard, which uses AWS) onto raw AWS and saved about $2500/month. You can do it too!

But... you do. If you get big and need to scale, you're going to want to take the control back. Them's the breaks.They should go dedicated for now, it's too early to colo IMO.

I don't know, $20k/mo strikes me as an awful lot of money to avoid engaging in work that a scrappy internet startup really ought to be competent at. If you don't know how to run the pipes and they get clogged and your plumber's not picking up the phone, you're screwed.That amount buys a whole lotta dedicated servers and the talent to run them. (Sidenote: Every time I price AWS or one of its competitors for a reasonably busy site, my eyes seem to pop out at the cost when compared to dedicated hardware and the corresponding sysadmin salary.)The larger issue is: Invest in your own sysadmin skills, it'll pay off in spades, especially when your back's up against the wall and you figure out that the vendor-which-solves-all-your-problems won't.

Because the whole point is that you shouldn't have to.

Why not hire a devops guy &#38; rack your own hardware? Or get some massive computing units at amazon (just as good but more expensive)?This reminds me of the excellent 5 stages of hosting story shared on here from a while back:<a href="http://blog.pinboard.in/2012/01/the_five_stages_of_hosting/" rel="nofollow">http://blog.pinboard.in/2012/01/the_five_stages_of_hosting/</a>

Not necessarily. Heroku claims that a global request queue is hard to scale, and therefore they switched to random load balancing. The comment above shows that a global request queue is not necessary. Lets say that minimum-of-n scales up to 10 dynos. If your application requires 40 dynos, you can have one front load balancer which dispatches the requests to 4 back load balancers, each of which has 10 dynos assigned to them on which they perform min-of-10 routing. This gives you routing that's almost as good as min-of-40 but it scales up nonetheless.

Assuming by intelligent, you mean minimally loaded, choice of 2 requires less bookkeeping than this. (choice of 2 requires only local (per-node level) info, where as most other intelligent load-balancing requires global info ie. the min, etc.)Taking the case of minimally loaded, you need to keep track of how many active requests each node/replica is serving, as well as globally keeping track of the min. (which past a certain load, will suffer a lot of contention to update)To do choice of 2, all you need is to keep track of active requests per node/replica.Under spiky workloads, there is also a problem with choosing minimally loaded. The counter for numRequests of a node might not update fast enough, so that a bunch of requests will go to that node, quickly saturating its capacity.Choice of 2 doesn't suffer this problem bec of its inherent randomization.

This is a knee-jerk reply.  I know, because my knee jerked as well.  Think about the problem a little more: if you have the data necessary to pick the min-of-two, then you have the data you need to do intelligent routing.

randomized routing is not necessarily bad if they look at 2 choices and pick the min. See <a href="http://en.wikipedia.org/wiki/2-choice_hashing" rel="nofollow">http://en.wikipedia.org/wiki/2-choice_hashing</a> and <a href="http://www.eecs.harvard.edu/~michaelm/postscripts/handbook2001.pdf" rel="nofollow">http://www.eecs.harvard.edu/~michaelm/postscripts/handbook20...</a>

Heroku implements this change in mid-2010, then sells to Salesforce six months later. Hmm...wondering how this impacted revenue numbers as customers had to scale up dynos following the change...

It's not exactly so, if you use a server that spawns child processes: <a href="http://michaelvanrooijen.com/articles/2011/06/01-more-concurrency-on-a-single-heroku-dyno-with-the-new-celadon-cedar-stack/" rel="nofollow">http://michaelvanrooijen.com/articles/2011/06/01-more-concur...</a> you can potentially handle 3-4 requests per dyno at a time. That doesn't fix the root problem, though.

That makes more efficient use of the nodes, because otherwise idle cpu time and memory allocation within a node has a chance of getting used, but it doesn't stop the queueing problem (it mearly gives you more virtual nodes to hand tasks to unintelligently).Also as the nodes are virtual machines anyway and may be contending with each other for IO, and for most apps these days you spend more time waiting for IO than you do spinning the CPU (unless you have a lot of static content so don't need to hit the db for many requests - but such requests are better handled by a caching layer above that which handles the fancier stuff), so the benefit of running multiple processes per node is going to be a lot less noticable than if you are talking about the nodes being physical machines with dedicated storage channels.

Is it so that a dyno can only handle a single user request at a time?Why dos it not use some kind of scheduling system to handle other task while one task is waiting on i/o?

&#62; Easiest fix: moving to EC2 next week. I've wanted to ever since these issues became evident but it's hard to make a good argument from handwaving about 'problems'.Of course, then you need to solve all these problems yourself. That sounds pretty easy, you'll have it done next week no problem!That was sarcastic, but this isn't: good luck, let us know how it goes.

Several Rails apps I develop have been suffering from similar issues. Perhaps 2-3% of requests take 0.4-2s in just processing. If the allocation is a little intelligent, it'll not perform too badly and is less work than much harder optimization. Yet if it's random, it'll queue up horribly.I'm pissed. Spent way too much time unable to explain it to coworkers, thinking I just didn't understand Heroku's platform and that it was my fault.Turns out, I didn't understand it, because Heroku never thought to clearly mention something that's pretty important.Easiest fix: moving to EC2 next week. I've wanted to ever since our issues became evident but it's hard to make a good argument from handwaving about 'problems'.

I work on New Relic's ruby agent, and you're right (hey Justin). In fact we support both methods (i.e. passing a timestamp or a duration). We rely on the front end server (e.g. nginx, apache) to set a timestamp in an HTTP header and forward that to the ruby application. In the case of heroku there is a special header that they pass which describes the queuing duration. Because we're in the ruby application we don't have control over whether this timestamp is accurate but I'm very interested in ideas on how we could do a better job in this situation.We do provide javascript based browser instrumentation ("Real User Monitoring") which measures request time from the browser's perspective. This might give you a more accurate idea of what real users are experiencing in this case.

It's interesting, because initially the way that queue time detection worked within New Relic was via timestamps.Currently, though, I believe it's just fed as a number of milliseconds: <a href="https://github.com/newrelic/rpm/blame/master/lib/new_relic/agent/instrumentation/queue_time.rb#L90" rel="nofollow">https://github.com/newrelic/rpm/blame/master/lib/new_relic/a...</a>This solves the issue of the application seeing out-of-whack queue times if there's clock skew between the front-end routing framework and the actual dyno box, but misses all the queued time spent in the dyno-queue per rap genius's post.

Best thing we can do is follow through on the article's call-to-action for emailing support@heroku.com:"After reading the following RapGenius article (<a href="http://rapgenius.com/James-somers-herokus-ugly-secret-lyrics" rel="nofollow">http://rapgenius.com/James-somers-herokus-ugly-secret-lyrics</a>), we are reevaluating the decision to use Heroku. I understand that using a webserver like unicorn_rails will alleviate the symptoms of the dyno queuing problem, but as a cash-strapped startup, cost-efficiency is of high importance.I look forward to hearing you address the concerns raised by the article, and hope that the issue can be resolved in a cost-effective manner for your customers."

Wow. This is explains a lot.We've always been of the opinion that queues were happening on the router, not on the dyno.We consistently see performance problems that, whilst we could tie down to a particular user request (file uploads for example, now moved to S3 direct), we could never figure out why this would result in queuing requests given Heroku's advertised "intelligent routing". We mistakenly thought the occasion slow request couldn't create a queue....although evidence pointed to the contrary.Now that it's apparent that requests are queuing on the dyno (although we have no way to tell from what I can gather) it makes the occasional "slow requests" we have all the more fatal. e.g. data exports, reporting and any other non-paged data request.

My company switched off of Heroku for our high-load app because of these same problems, but I still really like Heroku for apps with smaller loads, or ones in which I'm will to let a very small percentage of requests take a long time.

Wow.Normally when I read "X is screwing Y!!!" posts on Hacker News I generally consider them to be an overreaction or I can't relate. In this case, I think this was a reasonable reaction and I am immediately convinced never to rely on Heroku again.Does anyone have a reasonably easy to follow guide on moving from Heroku to AWS? Let's keep it simple and say I'm just looking to move an app with 2 web Dynos and 1 worker. I realize this is not the type of app that will be hurt by Heroku's new routing scheme but I might as well learn to get out before it's too late.

&#62; With many routers, intelligent routing has its own performance cost, the routers need to somehow synchronize state, and the simulation ignores this cost.Which is why we pay companies like Heroku to engineer clouds in which to run our applications. Because they're supposed to be better at this than us and spend the time and money building this difficult infrastructure well. That includes a scalable, stateful intelligent routing service.

I think this analysis and simulation does not account for one important thing: random routing is stateless and thus easy to be distributed. Routing to the least loaded Dyno needs to be stateful. It is quite easy to implement when you have one centralized router, but for 75 dynos this router would likely become a bottleneck. With many routers, intelligent routing has its own performance cost, the routers need to somehow synchronize state, and the simulation ignores this cost.

gooseGrade! I remembered the name. Here's a link: <a href="http://www.crunchbase.com/company/goosegrade" rel="nofollow">http://www.crunchbase.com/company/goosegrade</a>.

It's been tried before (Apture and a crowd-sourced proof-reading plug-in I can't remember the name of). It needs critical mass to work, but it might very well on a very community-focused platform.

Somewhat unrelated:Does anyone else think that RapGenius makes a great blogging platform? I'd love a plugin that enabled similar annotations on any blog, even if they're just by the original author and not crowdsourced.

Looks like Cloud 66 couldn't have picked a better day to announce their service: <a href="http://news.ycombinator.com/item?id=5213862" rel="nofollow">http://news.ycombinator.com/item?id=5213862</a>.

I don't think that thin will help in this case because Rails is blocking in general. So, you are right because other IOs will still block.You probably need an app that is built on like: <a href="https://github.com/raggi/async_sinatra" rel="nofollow">https://github.com/raggi/async_sinatra</a>

Question from a non-Ruby-expert: does Thin, which uses Event Machine, help with this at all, or do requests still block on other IO like database calls, etc?

I'd guess the problem wouldn't be as bad if each instance could handle more connections/requests than one or two. Allow say, 10 of them, and you will reduce the problem by a lot I believe.

How do they know what algorithm Heroku uses for randomization to stimulate the results?The differences in simulations are astonishing, I would not think Heroku's engineers were fine with this approach.'Let's push this random balancing out.. 1000% increase in resources? Oh well, just update documentation!'

Sure, but the real issue the article identifies is that, under random routing, they need to keep doubling the number of dynos to halve the odds of bad queueing, which leads to absurd factor of 50 requirements to get back to what they had before.  With round robin, the increase should be much more linear.

Nope, requests could still get queued behind a dyno that's busy with a long request.

For this specific issue round robin isn't really any different from random routing. Anything that stands a good chance of queueing a request while other machines sit idle is going to be a problem.The only way that round robin would be arguably better than random routing is if your random selection is not evenly distributed.

Maybe this is a dumb question, but wouldn't straightforward Round Robin routing by Heroku restore their "one dyno = one more concurrent request" promise without incurring the scaling liabilities of tracking load across an arbitrarily large number of dynos?

sudo for the user doing the deploy but not for the user running the app. The default is to run as the deploy user but you can change it.

Doesn't that require root or at least sudo permissions for your deploy user?

For those of you looking to migrate to other, barer hosting solutions like AWS or another VPS provider, I've put together a Capistrano add-on that let's you use Heroku-style buildpacks to deploy with Nginx doing front-end proxy. I use it for half a dozen apps on my VPSs and it works swimmingly well.<a href="https://github.com/peterkeen/capistrano-buildpack" rel="nofollow">https://github.com/peterkeen/capistrano-buildpack</a>

rap genius is definitely something bigger than rap lyrics. it could potentially be used for a platform for collectively deciphering any text.<a href="http://rapgenius.com/Marc-andreessen-why-andreessen-horowitz-is-investing-in-rap-genius-lyrics" rel="nofollow">http://rapgenius.com/Marc-andreessen-why-andreessen-horowitz...</a>

Funny - I thought this was a really interesting article - but I couldn't stand all these annotations.
And when I was selecting text (what i do mindlessly when I am reading an article) all the hell broke loose and it tried to load something.Very annoying when I want to concentrate on the technical details. So we'll see once again - everyone's different.

Aside from the Heroku issue, this is an amazing use of RapGenius for something besides rap lyrics. I didn't have to google anything in the article because of the annotations.

Why? The second is idling until the first fails, then the second takes over. Unless both fail simultaneously of course, for example due to power outage, but then your 40 servers will also fail simultaneously. Not to mention that with just one running server there are a lot fewer failure scenarios.

And when AWS goes down, your heroku dynos go tits up

Because when that one beefy server goes tits-up, you're out of business. Same with two.

Why not get yourself ONE beefy server? (or two) That should be able to handle your 150 requests per second, simplify your architecture a lot, and buying it would be cheaper than 1 month on Heroku (at $20,000/month).

Isn't this kind of a step backwards? Is Rails really that great that people are willing endure these kinds of limitations just to use it?

It is how Rails server behaves in itself, but that is also how Heroku tells you to do it.Rails can be served with Unicorn ( <a href="http://unicorn.bogomips.org/" rel="nofollow">http://unicorn.bogomips.org/</a> ) which is a forking app-server.I do believe there was a trick a while back where you could get Heroku to run a Unicorn process on a dyno to get more requests out of it. The process is described here: <a href="http://blog.codeship.io/2012/05/06/Unicorn-on-Heroku.html" rel="nofollow">http://blog.codeship.io/2012/05/06/Unicorn-on-Heroku.html</a>

Rails is commonly run as one or more application servers behind an http server that proxies requests to them. Rails itself doesn't manage threads or forked processes for accepting requests, so the only way it fits into Heroku's dyno model is as an app server per dyno.&#62; What are the advantages of this over standard server fork()/threaded accept designs?It's simple to build and manage in that you don't have to worry about thread safety and can use the already built and tested proxy capabilities of existing web servers to distribute traffic.

I assume it is intended to keep the nods as simple as possible so easy to schedule.A node dies? Just kill it and you lose at most the one active request.Which is the least busy node to send th enext request to? That can be a lot harder to judge reliably than simply "any nodes doing nothing? I'll queue this request then".

"For a Rails app, each dyno is capable of serving one request at a time."Is this a deliberate design choice on Heroku's part, or is this just how Ruby and Rails work? It sounds bizarre that you would need multiple virtual OS instances just to serve multiple requests at the same time. What are the advantages of this over standard server fork()/threaded accept designs?

I assumed people were running multiple rails processes on their dynos.<a href="http://michaelvanrooijen.com/articles/2011/06/01-more-concurrency-on-a-single-heroku-dyno-with-the-new-celadon-cedar-stack/" rel="nofollow">http://michaelvanrooijen.com/articles/2011/06/01-more-concur...</a>

The cost of New Relic on Heroku looks really high because each dyno is billed like a full server, which makes it many times more expensive than if you were to manage your own large EC2 servers and just have multiple rails workers.New Relic could be much more appealing if they had a pricing model that was based on usage instead of number of machines.

As long as they've got a limit on the maximum amount of concurrent requests, they'll be affected.  It might well not be as obvious.

For python / Django you can use a Procfile specifying gunicorn (rather than stock manage.py) with multiple worker threads, eg.web: gunicorn myapp.wsgi -b 0.0.0.0:$PORT -w 5Then you will have 5 parallel "single-threaded" instances on each dyno rather than just 1. This will partially ameliorate the problem, but probably not 100%. (NOTE: This is speculation since Heroku hasn't weighed in yet)

Does anyone know how if python applications are affected by this? I know they can handle multiple requests per dyno, I would be interested to know if random routing affects python apps too.

<a href="http://stackoverflow.com/questions/6370479/heroku-cedar-slower-response-time-than-bamboo" rel="nofollow">http://stackoverflow.com/questions/6370479/heroku-cedar-slow...</a> (from june 2011), the first discovery of cedar stack being slower than the bamboo stack

Intelligent routing would presumably need to know and act on a lot of state across their cluster, and if that's got to flow through a single node, you can see how it would present a bottle-neck as the cluster size and requests per second increased.On the other hand, you can do randomised routing without knowing any state at all. You can do it with more than one routing node as well, which makes scaling almost trivial.I presume there are Hard Problems associated with partitioning a Heroku-style cluster for intelligent routing, or that's what they would have done.

Statelessness. You don't have to remember where you've sent recent requests, which ones are still in process, or how long they've taken.

It's easier/cheaper for them to maintain and you have to pay for more dynos.  A lot more.

What's the advantage of randomized routing over intelligent routing? Why would this change be made?

We did this, but all it did was buy us a bit of extra time before we ran into the same problem again - a very small percentage (&#60;0.1%) of requests creating a queue that destroyed performance for the rest of them. Also, FWIW, Heroku does not officially support Unicorn, and you have to make sure that you don't run out of memory on your dynos (we tanked our app the first time we tried Unicorn with 4 processes).

OK, maybe I'm missing something here, but it seems to me that the OP's real problem is that he's artificially limiting himself to one request per dyno. They now allow a dyno to serve more than one request at a time, and he's presenting that as a bad thing! It seems to me that the answer to Rap Genius' problems is not "rage at Heroku," but rather "gem 'unicorn'".

This is how I run my apps as well, and they seem to handle more than one request concurrently per dyno, but I'm not smart enough to dispute this post, so I'm just sitting back and watching.

How I have been doing it for the last year.Puma define 4:8 threads or Unicorn 3 workers.

That's exactly what happened to us - switching to unicorn bought us a little time and a bit of performance, but we hit the exact same problems again after a couple more weeks of growth.

Yeah, the only real question is whether or not it's true that they no longer do intelligent routing. If that is the case, then regardless of anything else the problem exists once you pass a certain scale/request cost. It won't matter if that one dyno can handle hundreds of requests at once, it will still queue stupidly.

This is true - unicorn masks the symptoms for a period of time but does not solve the underlying problem in the way a global request queue would.Also, if the unicorn process is doing something cpu intensive (vs waiting on a 3rd party service or io etc) then it won't serve 3 requests simultaneously as fast as single processes would.

how does it decide to queue at the dyno level anyway?  Does it check for connection refusal at the TCP level?

That gets pretty unlikely, especially if you have many dynos and a low frequency of slow requests. The main reason unicorn can drastically reduce queue times here is that it does not use random routing internally.

Maybe this is a stupid question, but with unicorn it forks the request and can process multiple requests at the same time. Previously it seems that only one request could be handled by the dyno so requests had to queue on the dynamic routing layer but with multiple request support with unicorn or whatever, wouldn't it be more efficient to dump all the requests to dynos?  Followup question, also how would intelligent routing work if it just previously checked to see if which dyno had no requests?  That seems like an easy thing to do, now you would have to check CPU/IO whatever and route based on load.  Not specifically targeted at you but to everyone reading the thread.

If you have 2 unicorn servers and you happen to get 3 slow requests routed to it, you are still screwed, right? Seems to me like it will still queue on that dyno.

Nevertheless, random routing is a bad idea even if a dyno can handle multiple requests simultaneously

Right if the apps internal queue is full and it stops accepting connections I'm assuming it will still queue at the dyno level anyway.

Yes, it's true that the Cedar stack supports forking web servers like unicorn, and that an individual dyno can run multiple workers and therefore serve multiple requests at the same time.However, dumb routing is still very problematic – even if your dyno can work on two requests simultaneously it's still bad for it to get sent a third request when there are other open dynos.Also, for apps with a large-ish memory footprint, you can't run very many workers. A heroku dyno has 512mb memory, so if your app has a 250mb footprint, then you can basically only have two workers.Another essential point to note is that the routing between cedar and bamboo is essentially unchanged. They simply changed the type of apps you can run.

I have been using it on a few small projects. Haven't ran into an issue yet and the setup has been really easy.

I did some performance analysis on puma vs unicorn vs thin a while ago:<a href="http://ylan.segal-family.com/blog/2012/08/20/better-performance-on-heroku-thins-vs-unicorn-vs-puma/" rel="nofollow">http://ylan.segal-family.com/blog/2012/08/20/better-performa...</a>Although as noted in the comments, I neglected to run threadsafe! and should have probably tried rubinius or jruby. I have been meaning to redo. Take with a grain of salt

We are running a smaller app on JRuby using Puma with thread safe on. It has a significantly smaller footprint, as we are only booting one server.Overall really solid, though more useful if you can use something other than MRI.

I don't have a stake in the Ruby webserver wars, but the unicorn site has a very good discussion about how it works internally, why it's build as it is, pros &#38; cons etc..This seems to be missing from most of these project sites, which are often just marketing (look! It's better!!), and therefore not very trustworthy.From the outside it looks like the biggest differentiator in each generation of ruby servers (and, I guess, db managment systems :) is not that the new is better or worse, but simply that has different trade-offs.

Slightly off topic, but what are everyone's. experience and thoughts about Puma[1]?I am using it on a small production environment with Heroku and I like it, but when we officially launch the app, should we switch to Unicorn?[1] <a href="http://puma.io/" rel="nofollow">http://puma.io/</a>

You don't happen to have any documentation for how to do that do you? Very curious. Never seen anything about setting up Unicorn like this prior (I'm just using 3 or 4 Unicorns/dyno currently)

To amplify, I do a ton of queue adjustment work with Unicorn at the backlog level. It is so frequent that we set up Unicorn startup to read from an ENV variable on Heroku that we set as needed.With two Unicorn workers we found that 25 was the best backlog threshold to accept (it refuses additional requests). When we were able to go to 5 Unicorn workers on Heroku we had to start to adjust that.

I'm inclined to wait until Heroku weighs in to render judgement. Specifically, because their argument depends on this premise:&#62; But elsewhere in their current docs, they make the same old statement loud and clear: 
&#62; The heroku.com stack only supports single threaded requests. Even if your applicaExplaintion were to fork and support handling multiple requests at once, the routing mesh will never serve more than a single request to a dyno at a time.They pull this from Heroku's documentation on the Bamboo stack [1], but then extrapolate and say it also applies to Heroku's Cedar stack.However, I don't believe this to be true. Recently, I wrote a brief tutorial on implementing Google Apps' openID into your Rails app.The underlying problem with doing so on a free (single-dyno) Heroku app is that while your app makes an authentication request to Google, Google turns around and makes a "oh hey" request to your app. With a single-concurrency system, Google your app times out waiting for Google to get back to you and Google won't get back to you until your app gets back to you so hey deadlock.However, there is a work-around on the Cedar stack: configure the unicorn server to supply 4 or so worker processes for your web server, and the Heroku routing mesh appropriately routes multiple concurrent requests to Unicorn/my app. This immediately fixed my deadlock problem. I have code and more details in a blog post I wrote recently. [2]This seems to be confirmed by Heroku's documentation on dynos [3]: 
&#62; Multi-threaded or event-driven environments like Java, Unicorn, and Node.js can handle many concurrent requests. Load testing these applications is the only realistic way to determine request throughput.I might be missing something really obvious here, but to summarize: their premise is that Heroku only supports single-threaded requests, which is true on the legacy Bamboo stack but I don't believe to be true on Cedar, which they consider their "canonical" stack and where I have been hosting Rails apps for quite a while.[1] <a href="https://devcenter.heroku.com/articles/http-routing-bamboo" rel="nofollow">https://devcenter.heroku.com/articles/http-routing-bamboo</a>[2] <a href="http://www.thirdprestige.com/posts/your-website-and-email-accounts-should-be-friends-part-ii" rel="nofollow">http://www.thirdprestige.com/posts/your-website-and-email-ac...</a>[3] <a href="https://devcenter.heroku.com/articles/dynos#dynos-and-requests" rel="nofollow">https://devcenter.heroku.com/articles/dynos#dynos-and-reques...</a>[edit: formatting]

If it's true, I can't see how random routing can be anything but a cynical cash grab.Even a very simple algorithm like round robin would give you a significantly better latency characteristic wouldn't it?

The problem is the request arrival rate vs the distribution of service times in your app.New Relic may be giving you an average number you feel happy about, but the 99th percentile numbers are extremely important. If you have a small fraction of requests that take much longer to process, you'll end up with queuing, even with a predictive least loaded balancing policy.This is a very common performance problem in rails apps, because developers often use active record's associations without any sort of limit on row count, not considering that in the future individual users might have 10000 posts/friends/whatever associated object.Fix this and you'll see your end user latency come back in line.

Thanks for the write up.  I've been looking for some more reviews on Heorku's platform and this in-depth review definitely illuminates some challenges with the platform.

I'm glad you clarified that your first sentence is about their CSS choices. ;)

I think that speaks more on your dormant racism than anyone elses.

Perspective is a hell of a thing....The way this comment reads, I thought this was going to get racist real quick - but was relieved when I finished reading and did agree with you :)

For this blog-post where the annotations aren't such a big deal the readability bookmarklet[1] works quite well.[1] <a href="http://readability.com/bookmarklets" rel="nofollow">http://readability.com/bookmarklets</a>

I really like Rap Genius, but I wish they would tone down the blackness of the background. Reading #CCC text on #000 background makes my eyes bug out.

Most of the 'Power of two choices' I've read about assumes the presence of a global queue (<a href="http://www.eecs.harvard.edu/~michaelm/postscripts/handbook2001.pdf" rel="nofollow">http://www.eecs.harvard.edu/~michaelm/postscripts/handbook20...</a>) -- there's a parallel variety, but they go light on details in that text.I'm unaware of how Heroku does things. I'd guess they dropped the global queue because it's unpractical (failure prone, not scalable as it's a single point of contention).I'm mostly surprised to see people happy being able to handle 1 or 2 requests in parallel per instance in general. That sounds absolutely insane to me.

On further thought, Heroku users could probably even 
approximate the benefits from the Mitzenmacher power-of-two-choices insight (mentioned elsewhere in thread), without Heroku's systemic help, by having dynos shed their own excess load.Assume each unicorn can tell how many of its workers are engaged. The 1st thing any worker does – before any other IO/DB/net-intensive work – would be to check if the dyno is 'loaded', defined as all other workers (perhaps just one, for workers=2) on the same dyno already being engaged. If so, the request is redirected to a secondary hostname, getting random assignment to a (usually) different dyno.The result: fewer big pileups unless completely saturated, and performance approaching smart routing but without central state/queueing. There is an overhead cost of the redirects... but that seems to fit the folk wisdom (others have also shared elsewhere in thread) that a hit to average latency is worth it to get rid of the long tails.(Also, perhaps Heroku's routing mesh could intercept a dyno load-shedding response, ameliorating pile-ups without taking the full step back to stateful smart balancing.)Added: On even further thought: perhaps the Heroku routing mesh automatically tries another dyno when one refuses the connection. In such a case, you could set your listening server (g/unicorn or similar) to have a minimal listen-backlog queue, say just 1 (or the number of workers). Then once it's busy, a connect-attempt will fail quickly (rather than queue up), and the balancer will try another random dyno. That's as good as the 1-request-per-dyno-but-intelligent-routing that RapGenius wants... and might be completely within RapGenius's power to implement without any fixes from Heroku.

They want to force the issue with a public spat. Fair enough.But, they also might also be able to self-help quite a bit. RG makes no mention of using more than 1 unicorn worker per dyno. That could help, making a smaller number of dynos behave more like a larger number. I think it was around when Heroku switched to random routing that they also became more officially supportive of dynos handling multiple requests at once.There's still the risk of random pileups behind long-running requests, and as others have noted, it's that long-tail of long-running requests that messes things up. Besides diving into the worst offender requests, perhaps simply segregating those requests to a different Heroku-app would lead to a giant speedup for most users, who rarely do long-running requests.Then, the 90% of requests that never take more than a second would stay in one bank of dynos, never having pathological pile-ups, while the 10% that take 1-6 seconds would go to another bank (by different entry URL hostname). There'd still be awful pile-ups there, but for less-frequent requests, perhaps only used by a subset of users/crawler-bots, who don't mind waiting.

If this is such a problem for you, why are you still on Heroku? It's not a be-all end-all solution.I got started on Heroku for a project, and I also ran into limitations of the platform. I think it can work for some types of projects, but it's really not that expensive to host 15m uniques/month on your own hardware. You can do just about anything on Heroku, but as your organization and company grow it makes sense to do what's right for the product, and not necessarily whats easy anymore.FYI I wrote up several posts about it, though my reasons were different (and my use-case is quite a bit different from a traditional app):* <a href="http://justcramer.com/2012/06/02/the-cloud-is-not-for-you/" rel="nofollow">http://justcramer.com/2012/06/02/the-cloud-is-not-for-you/</a>* <a href="http://justcramer.com/2012/08/30/how-noops-works-for-sentry/" rel="nofollow">http://justcramer.com/2012/08/30/how-noops-works-for-sentry/</a>

While I generally agree with your thoughts, I also wonder what's the reason for continuing to misrepresent their service until you dig 20 layers deep in docs.

We were very surprised to discover Heroku no longer has a global request queue, and spent a good bit of time debugging performance issues to find this was the culprit.Heroku is a great company, and I imagine there was some technical reason they did it (not an evil plot to make more money). But not having a global request queue (or "intelligent routing") definitely makes their platform less useful. Moving to Unicorn helped a bit in the short term, but is not a complete solution.

Interesting article but every time I read something on RapGenius and move my eyes from the screen, I keep seeing white lines, very annoying. White font on black background is bad. Off topic I know, but still.

this is a bit hyperbolic. "Heroku Swindle Factor" just seems rude.

I believe routing is not random, but round robin. I'd like Heroku to confirm. It's still a problem. If you are looking to run Unicorn on Heroku, use the heroku-forward gem (<a href="https://github.com/dblock/heroku-forward" rel="nofollow">https://github.com/dblock/heroku-forward</a>). Works well, but application RAM is quickly its own issue, we failed to run that in production as our app takes ~300MB.

I was thinking the same thing.  And 200ms for what exactly?  To me the elephant in the room is that 99 server instances are spun up to handle 15M uniques per month.

Single request per server?  What year is this?  2003?

no, at x level of concurrency most set ups wont spend 16 seconds or more on 10% of their requests.

Well, at X level of concurrency, wouldn't most set ups with load balancers start to spit numbers like that?

High cost/risk associated with switching providers, and frog-in-heating-water syndrome.

&#62; But still WTF. 89/100 requests failed? That's not happy-making.Unless something has changed recently, ab doesn't handle dynamic pages very well. It takes the first pageload as a baseline, and any subsequent request with any portion of the page that is randomized, or is a CSRF token, or reflects most recent changes, etc., is marked as "failed" because it doesn't match the baseline's length.The page in question does have a block in the footer reflecting "hot songs", which I'm guessing changed a bit during the run.

&#62; I've never understood using round-robin or random algorithms for load-balancers...LeastConns/FastestConn selection is very dangerous when a backend host fails. Imagine a host has a partial failure, allowing health checks to pass. This host now fast fails and returns a 500 faster than other hosts in the pool generate 200s. This poison host will have less active connections and your LB will route more requests to it. A single host failure just became a major outage.I like WRR selection for back ends, then use a queue or fast fail when your max active conns is exceeded. Humans prefer latency to errors, so let the lb queue (to a limit) on human centric VIPs. Automated clients deal better with errors so have your lb throw a 500 directly, or RST, or direct to a pool that serves static content.

Worse than that:<pre><code> * 89/100 requests failed (according to
 https://gist.github.com/a-warner/c8cc02565dc214d5f77d ). 
 * Heroku times out requests after 30 seconds, so the 30000ms
 numbers may be timeouts (I've forgotten if *ab* includes 
 those in the summary).
 * That said, the *ab* stats could be biased by using overly 
 large concurrency settings (not probably if you're running 50 dynos...),
 but still...
</code></pre>
But still WTF. 89/100 requests failed? That's not happy-making.Uncertainty is DiaI (death-in-an-infrastructure). I just created a couple of projects on Heroku and love the service, but this needs to be addressed ASAP (even if addressing it is just a blog post).Also, if you have fewest-connections available, I've never understood using round-robin or random algorithms for load-balancers...

Nodejitsu actually does have a free plan if your project is Open Source.<a href="http://blog.nodejitsu.com/opensource-dot-jitsu" rel="nofollow">http://blog.nodejitsu.com/opensource-dot-jitsu</a>

&#62; How many dynos does Heroku have running for non-paying customers? Do these free dynos actually necessitate this random routing mesh bullshit? If not, what?Of course the random routing mesh isn't necessitated by anything, this problem is already solved by bigger companies.

I've been scanning through the comments, and I have yet to see anything written by a Heroku engineer to defend the company. I'm hoping its in here and I missed it. I have a feeling that this all might be absolutely true, and they have lawyers/PR trying to think of a damage control plan.I suspect that the reason they'be been pushed to do this is financial, and it makes me think that Nodejitsu's model of simply not providing ANY free plans other than one month trials is a good one. I realize it's apples and oranges, since NJ is focused on async and this wouldn't even be a problem for a Node app, but from a business perspective I feel like this would alleviate pressure. How many dynos does Heroku have running for non-paying customers? Do these free dynos actually necessitate this random routing mesh bullshit? If not, what?

Good lord!!!!!Percentage of the requests served within a certain time (ms)<pre><code> 50% 844

 66% 2977

 75% 5032

 80% 7575

 90% 16052

 95% 20069

 98% 29282

 99% 30029

 100% 30029 (longest request)
</code></pre>
Those numbers are amazingly awful. If I ever run ab and see 4 digits I assume I need to optimize my software or server. But 5 digits?Why in the world would a company spend $20,000 a month for service this awful?

Take a look at deliver if you like heroku push, but want it on a machine you control at a bit lower level: <a href="https://github.com/gerhard/deliver" rel="nofollow">https://github.com/gerhard/deliver</a>
I got it working on EC2 / Ubuntu real easy, and even added some basic support for SmartOS/illumos for joyent cloud

Thanks for calling this out. As you said, random routing is about as naive as it gets. They need to make upgrades to the routing mesh - expose some internal stats about dyno performance and route accordingly. Even if the stats were rudimentary, anything would be an improvement over random.

Can't they make the choice of intelligent or random scheduling a per-platform setting?Java, node.js use random. Django + Rails use intelligent.

I'd been using Heroku since forever, but bailed on them for a high traffic app last year (Olympics related) due to poor performance once we hit a certain load (adding dynos made very little difference). We were paying for their (new at the time) critical app support, and I brought up that it appears to be failing at a routing level continuously. And this was with a Sinatra app served by Unicorn (which at the time at least was considered unsupported).We went with a metal cluster setup and everything ran super smooth. I never did figure out what the problem was with Heroku though and this article has been a very illuminating read.

I would like to see why actually Heroku fell back to random routing. It doesn't really make sense. Of course this all routing stuff is really tricky, but on the other hand there is a lot of work done (look at TCP agorithms). 
When I was studying ZeroMQ routing based stuff for one project, I came across "credit-based flow control" pattern, that could make perfect sense in this kind of situation (Publisher-Subscriber scenario). Why not implementing such thing?

The balance of a subjective, sensationalist headline with objective statistical simulation was impressive.I'm a huge Heroku fan using Cedar/Java, but can't help but wonder how many optimization options remain for Rails Developers, assuming nothing else changes on Heroku:* Serving static HTML from CDN
* Unicorn
* Redis caching with multiget requests

&#62; If requests are getting distributed randomly shouldn't all your pages show a similar average time in queue?A common misconception, called "the law of small numbers".Probability theory tells us this is only true over a large amount of requests, i.e. in the long term (the law of large numebrs). In the short term, results can vary wildly and thus form these kind of queues.

You don't think that page was chosen as an example because it should be a very inexpensive page to render? I read that as saying that the performance problem does affect all pages "even on THIS mostly static page."As for your skepticism of the graphs, take a look at the annotated R source they provided. I didn't do a deep dive on it or anything but it looked reasonable to me.

Those charts of "simulated" load balancing strategies don't look at all reasonable at first glance. You certainly don't see such spiky patterns with normal web loads. I think you'd have to have some crazy amount of std. dev in completion time cranked way, way up in your simulation before you saw a bunch of servers stacked at 30 with others at 1.It's not that there is no benefit to better balancing, it's just that I've never seen it have anything close to that impact. It seems like it's only being perceived as a problem here because somebody drank too much of the (old) kool-aid.Some of the other numbers are hard to take at face value as well. 6000ms avg on a specific page? If requests are getting distributed randomly shouldn't all your pages show a similar average time in queue? Sounds more like they're using a hash balancing alg and the static page was hashing on to a hot spot.

This really sux. I like all their other offerings though - I'm considering running the Cloud Foundry "dyno" part alone and using the heroku services with it.

I believe Mongrel2 (<a href="http://mongrel2.org" rel="nofollow">http://mongrel2.org</a>) is close to what you're talking about. It uses zeromq (<a href="http://www.zeromq.org" rel="nofollow">http://www.zeromq.org</a>) to talk to a backend application and the backend application talks back to the web server using the same protocol.Not exactly nodes pulling messages off a queue, but closer to something like that?

In Mongrel2, app servers subscribe to a named ZeroMQ queue and, when they're done, they send the response on a different queue.You can actually configure an arbitrary number of different queues if you like, switching on request path and some other stuff I don't just now recall.

The pull model is very hard to implement because the router behaves like a proxy for a much larger set of dynos (think tens of thousands). When you have 10K clients yielding "i'm available" 10 times a second, you have a nightmare, it's not sustainable.A possible solution for the proxy and the dynos to agree on a protocol where the proxy passes a request to the dyno and the latter can give up with a status code that says "retry with another dyno". This could go on to up to the 30s timeout limit that Heroku has now.

This kind of validates an idea I've been flirting with: a Heroku-like service which routes requests via AMQP or similar message broker and actually exposes the routing dynamics to the client apps.From a naive, inexperienced view the idea of having web nodes "pull" requests from a central queue rather than the queue taking uneducated guesses seems to be a no-brainer. I can see this making long-running requests (keep-alive, streaming, etc) a bit more difficult, but not impossible.What am I missing? This seems so glaringly obvious that it must have been done before...

Heroku's Ugly Secret: The story of how the cloud-king turned its back on Rails