Speed improvements are always nice, but the big improvement for us came from moving from 1x and 2x dynos to Performance dynos. CPU performance on the 1x and 2x dynos was highly inconsistent, and would suddenly become abysmal for periods of 5-10 minutes or more - and due to random routing, the backed up dyno would keep on getting requests until the requests start timing out.
I've been impressed with the performance work you've been doing across the board. But in these sorts of write-ups (and the corresponding issue trackers), it'd be very helpful to qualify with which version of Ruby you're using. I've surmised it's MRI, but even then different MRI versions can have different performance profiles. It'd be helpful to know what your GC tuning looks like, too. Particularly in cases where object allocation is determined to be a factor.
Anyway, this is all good stuff and I don't doubt there's an improvement. A little expansion on the methodology would just make it all the more valuable.
Tangential note: In one of the commit comments there's an exchange about the performance of `any?` and it links to MRI source. However, it links to a line in master, so the link will eventually fall out of date. If you press 'y' in Github it'll change the URL to one with a SHA in it. Or maybe pick the release tag so the link doesn't change as the file updates.
> I've been impressed with the performance work you've been doing across the board
Thanks!
For all the tests I used MRI Ruby 2.2.2. Unless otherwise noted, I almost exclusively use the most recently released MRI Ruby version for testing. There shouldn't be a huge difference in most recent Ruby versions. The only thing that comes to mind is that `String#freeze` existed in older versions but was a method call, and not implemented as a parsing optimization (i.e. it would still allocate a string) however I believe that's been in there for a few years. Maybe the generational GC might have an impact.
> It'd be helpful to know what your GC tuning looks like, too.
I used stock GC settings for all the benchmarks
```
$ env | grep GC
# nothing here
```
> A little expansion on the methodology would just make it all the more valuable.
The patch does have some more specifics than the post, especially the commits. I used benchmark/ips in some of them to demonstrate specifics problems. I can do a larger write up on specific methodology, if you think that would be interesting.
> If you press 'y'
Thanks, i've been trying to get in the habit of doing so, that's a good tip.
> For all the tests I used MRI Ruby 2.2.2. Unless otherwise noted, I almost exclusively use the most recently released MRI Ruby version for testing.
Good to know. I guess I'm suggesting that specifically calling that out in the text will help it stand the test of time. Otherwise the reader needs to cross-reference the date with the MRI release list.
And really, this is all I was looking for in the methodology. The post covers what you did in pretty good detail. The GC thing was something I was curious about just because that's ping-ponged quite a bit throughout the 2.x releases.
maybe is a stupid and generic question, but I will ask it :)
using the current libraries, for searching way to improve rails performance, where to you think there are hot places where is easy to go and improve?
second one.
I know that there is plan for rack2, and some clean-up. Can we dream to a more faster rails5?
last one. Not related with your performance improvements.
I found most of the time that AR is quite slow, and I always tend to use ActiveRecord::Base.connection.execute("query here"). What is your opinion about http://sequel.jeremyevans.net/ ?
I think we're to the point where there's not many low hanging performance optimizations at the low level. Most apps that are really slow are slow because of their codebase and not Rails, that doesn't mean we shouldn't make Rails as fast as possible. Many of these optimizations I made here were found based on my apps use of the Rails api. I'm hoping others will benchmark their apps and look where we can save a few fractional milliseconds here and there. It's all the small speed-ups that make an overall faster product.
> I found most of the time that AR is quite slow
There are different APIs that are faster than others, i believe the `Model#find` where you pass in a primary key is pretty fast. I actually had a hard time finding any optimizable code in AR, MR. Patterson and Sean Griffin have done a great job of optimizing, though each codebase and use case is different. I've used sequel and really enjoy it. Jeremy writes some amazing code https://github.com/mime-types/ruby-mime-types/commit/8ce1add... and is a great human.
On the third point, my sense is that the part of AR that is often expensive is building full objects, when you often only need a few of their attributes. This is not such a big deal with a single (or a few) `find`s, but often is when you're grabbing lots of records, even when using `includes` properly. I find myself using `pluck` frequently when tracking down performance issues.
I often wish for a pared down database interaction API (I hesitate to call it an ORM) that just exposes raw data, and provides a convenient query interface (like AR does, but also like raw arel). Maybe this is what Sequel does, and I should be using that, I'm not sure. I was excited about datamapper2 and I've looked at ROM[0], but I'm not sure it's what I'm dreaming of. I could probably cobble it together by using existing database driver gems and arel directly. Unfortunately, as is usually the case when considering venturing off the beaten path of Rails, this sort of thing would give up a lot of the "conventions" advantages of Rails. AR pretty much is Rails.
That is, indeed, what Sequel does. You can query the database object directly (for example, `DB[:posts].where{comments_count > 5}.exclude(poster_name: "Bob").order(:comments_count.desc).limit(10).all`) and get back an array of hashes.
Sequel's model layer is also faster than ActiveRecord's, because much of the additional functionality that ActiveRecord piles onto all records (dirty tracking, single table inheritance, etc.) are available in Sequel via a plugin system. You can enable the plugins you want to use and not pay the overhead of all the others. You can even enable specific plugins for only the models where you'll actually want to use them.
It also has a lot less magic (no association proxies, unless you enable the plugin for them :)), ridiculously customizable (many more options for associations, custom eager loading logic, and so on), and has an implementation of its Postgres adapter written in C for performance.
Highly, highly recommended.
Edit: Oh, and an issue tracker that is almost always at zero, with a very fast response time. As someone who has contributed a patch to ActiveRecord, I can't tell you how nice that is.
Yeah, that does sound like exactly what I want. Still has the "conventions" problem I mentioned, but maybe worth it, and common enough that it isn't that big a deal.
To your edit: Ha, I have a PR against arel that is similarly neglected[0]. Maybe an advantage of using a less popular library is that the maintainers aren't so overwhelmed that things get lost in the shuffle!
On the third point, my sense is that the part of AR
that is often expensive is building full objects,
when you often only need a few of their attributes.
Puma's currently the recommended one, but I think that's actually a bad recommendation.
Puma is great if you have a Ruby implementation without a GIL (e.g. JRuby, which it was built for), but it's not so good on MRI, because you have to use clustered mode, which runs multiple processes. What they don't tell you is that when running clustered mode, the requests are randomly distributed to each process, and then the requests queued at that level. Each process can run multiple threads each processing a request simultaneously, but because of the GIL anything that's computationally heavy will cause the other requests in that process to run slowly.
That means that it's possible for one process to become backed up with requests, while the other processes on the dyno are sitting idle. And it's actually not just possible, but likely when under reasonable load.
So there is actually no deployment configuration of Puma which makes sense under MRI, at least without a reverse proxy on the server handling queuing:
- Single process (no clusters), 1 thread: Bad because you use only one core, but consistent performance assuming you're not on a multitenant 1X or 2X dyno.
- Clustered, 1 thread per process: Bad because requests are queued at the process level, so a slow request will cause any others sent to that process to have to wait. If you have a request which takes 25s to execute, the one sitting behind it in the queue is highly likely to timeout, and this can cascade to the next request, etc.
- Clustered, multiple threads per process: Better, but still bad, because while multiple requests will be running at the same time in the process, they are competing for the GIL and can slow each other down so much you'll still get timeouts.
The best solution we found to all this was to run a buildpack with a nginx reverse-proxy to a Unicorn server. It's nice because each request is run in it's own process, and the requests are queued at the dyno-level by nginx until any of the processes is free. Combine that with Performance dynos, so you can run a small number of dynos with lots of workers each, where each dyno has consistent performance, and the queuing issues mostly go away.
As you may be able to tell, I've spent far more time debugging this behavior than I would have liked to!
> Puma is great if you have a Ruby implementation without a GIL
Puma is also great if you have blocking IO like database queries or file uploads which most ruby/rails apps do. The GIL is released when Ruby does IO so it can context switch and run other threads.
> there is actually no deployment configuration of Puma which makes sense under MRI
I disagree, puma makes lots of sense on MRI but it is app specific. By this same argument Sidekiq would be worthless on MRI but it's the dominant work queueing library even though it is solely threaded. In my experience Puma with MRI using "hybrid" with clusters and threads is faster than Unicorn (or rather you can get more throughput and less queueing). However it's app specific. If your app is mostly doing number crunching in Ruby code and not touching IO, then the GIL never gets released and you don't gain much by using threads. Again most Ruby apps use a lot of IO.
> anything that's computationally heavy will cause the other requests in that process to run slowly.
Exactly.
One of the main reasons I pulled recommendation for Unicorn was it's lack of slow client handling, it doesn't help to have a fast webserver, if all your workers are...waiting...on...bytes...and...can't...do...anything...else. Nginx handles slow clients for you so nginx + unicorn is good, it's a harder deployment configuration and it's easier to tell our customers to configure one thing instead of 2, especially if that 1 thing yields better results for the majority of use cases.
I'm glad you got things working with nginx + unicorn. I'm a big fan of loading up PX dynos with lots of workers :) I hope some of the above helps clear up a few points for future readers. I always encourage people to try different options and take our recommendations with a grain of salt. It's also why we've kept our Unicorn article so that anyone not happy with puma performance can try another webserver. Thanks for sharing!
Thanks - I wrote this mostly as a braindump of some of the things I discovered in investigating this stuff - We had a period of several months where we were constantly getting timeouts. There's not really a huge amount of really detailed information out there on how the different servers work and the tradeoffs between them.
RE threads on MRI, I think it comes down to what your latency requirements are and the type of work as you said. I'm happy with sidekiq on MRI because it doesn't matter hugely if the process gets bogged down for 60 seconds while a ruby-heavy job runs and the other jobs run slowly - jobs are only pulled from redis when there is a spare thread, and sidekiq jobs won't timeout like web requests do. IO-heavy work will get a boost with threads on Puma, but if you get a single Ruby-heavy request running for 20 seconds serializing something, you are going to get timeouts happen.
I guess it does depend on the type of app you are running - Ours is actually quite Ruby-crunching heavy, because it has to serialize some big json responses, which is still incredibly slow in Rails. Apps which don't have occasional Ruby-intensive requests wouldn't have as much trouble with all this :)
A lot of time according to new relic is spent in Middleware/Rack/ActionDispatch::Routing::RouteSet#call on one of our Rails apps on Heroku. Any ideas on what might be the cause? (I'm within memory limits standard-2x)
Whooops responded with old account. If you run `$ env RAILS_ENV=production rake middleware` you'll see that your final app that gets mounted is actually the "routes" of your app.
run CodeTriage::Application.routes
Essentially that RouteSet#call wraps every piece of code you've ever written for your app. I think that newrelic is reporting that is the entry point to your app code which makes sense that it takes the longest amount of time.
Mutable strings are a double edged sword from a performance perspective. It sucks to have a bunch of un-needed string allocations due to string literals that are only used for one method. Sometimes though, mutating and re-using strings when possible can yield some pretty large improvements like https://github.com/mime-types/ruby-mime-types/pull/93.
I would love to read a post written in simple(ish) terms about how V8 handles string allocation and what string performance optimizations helped the most. Would be really neat to see it compared to python/ruby/etc.
It's also worth noting that the majority of the speed here comes from getting rid of hash and array allocations. The bulk of the object count came from string savings but the bulk of the "memory" came from more complex objects.
