Intro
Recently someone asked the question: “Where are the Grails vs Rails benchmarks??”. Since we actually haven’t got round to optimising Grails I thought it was silly that we should benchmark a 0.4.2 release against Rails which is now at 1.2.x. Nevertheless, to allay concerns about Grails I decided we should do it anyway. This page represents the results.
The reality of this benchmark is that it is actually more like Rails vs Groovy + Spring MVC + Hibernate + Sitemesh. Grails can take some credit of course, but the heavy lifting is being done by Spring & Hibernate in particular.
 |
I am by no means a benchmarking expert, so if you have any changes or recommendations please shout |
Update - After feedback from Jared Richardson who detailed how with Rails & Mongrel you actually need multiple Mongrel instances to be equivalent to tomcat I decided to conduct some further tests. This basically showed my naivety when configuring Rails, but there you go, so I apologise for the falseness of the original benchmarks.
To be clear what I have done is do configure Rails with a 10 Mongrel cluster and Pound as per Jared’s tutorial here. To make things fair I also reduced Tomcat to use only 10 Java threads. The results can be seen in the updated benchmarks that follow.
If I have done something wrong in the Rails configuration please point it out, because if you read on, doing the clustering didn’t help much.
Hardware Specs
We are trying to be entirely open about this benchmark. Personally I feel benchmarks are of limited value, but some live by them so there you go. To get some of the details out the way then. The test hardware is as follows:
Apple MacBook
1.83ghz Intel Core Duo
1GB 667 Mhz DDR2 SDRAM
Ok I know, I know it isn’t a server, but I didn’t have one lying around, so this will have to do. It has dual cores after all 
Software Specs
Now onto the software platforms:
Grails
- OS: Mac OS X 10.4.9
- Server: Apache Tomcat 5.5.20
- Version: 0.5-SNAPSHOT from 20th of March
- Environment: Production
- Database: MySQL 5.0.27
- Java: Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_07-164)
- JDBC Driver: mysql-connector-java-3.1.10-bin.jar
|
I decided to benchmark with 0.5-SNAPSHOT because it implements the new URL mapping feature and since Rails has routes it evens the playing field out a bit. In other words it would be unfair on Rails if we benchmarked with Grails not having a routing feature, as it does in the upcoming 0.5 |
Configuration A: Standard Tomcat
The first test was with a standard Apache Tomcat 5.5.20 configuration, with the following settings:
<Connector port=“8080″ maxHttpHeaderSize=“8192″
maxThreads=“150″ minSpareThreads=“25″ maxSpareThreads=“75″
enableLookups=“false” redirectPort=“8443″ acceptCount=“100″
connectionTimeout=“20000″ disableUploadTimeout=“true” />
Configuration B: Tomcat configured with a 10 thread pool
To even things up with Rails having 10 mongrel’s I decided to configure Tomcat with only 10 threads with the following configuration:
<Connector port=“8080″ maxHttpHeaderSize=“8192″
maxThreads=“10″ minSpareThreads=“1″ maxSpareThreads=“1″
enableLookups=“false” redirectPort=“8443″ acceptCount=“100″
connectionTimeout=“20000″ disableUploadTimeout=“true” />
Rails
Set-up according to the article here: http://hivelogic.com/narrative/articles/ruby-rails-mongrel-mysql-osx
- OS: Mac OS X 10.4.9
- Server: Mongrel 1.0.1
- Version 1.2.3
- Environment: Production (Started with mongrel_rails start -e production)
- Database: MySQL 5.0.27
- Other notes: I installed the Ruby MySQL native bindings as per the aforementioned article
So the same database for both and the most commonly deployed container for Grails (there might be faster containers out there, but I’m not here to test containers). As for Rails I have been told by multiple sources that Mongrel is THE container to deploy to for maximum performance, but I’m no expert so please correct me if I’m wrong here.
|
Also note that I have not bothered to connect Grails or Rails up to Apache using mod_jk or the equivalent Rails FastCGI connector. |
Configuration A: A single mongrel instance
The first Rails configuration was using only a single mongrel instance and hitting it directly on port 3000.
Configuration B: 10 mongrels with load balancing by Pound
The second configuration I used a 10 mongrel cluster configured as per Jared Richardson’s instructions here and configured it with this command:
sudo mongrel_rails cluster::configure -e production -p 8001 -N 10
To start-up the cluster I did:
sudo mongrel_rails cluster::start
I then used Pound as a load balancer with the following configuration again setup as per Jared’s instructions:
ListenHTTP
Address 0.0.0.0
Port 3000
End
Service
BackEnd
Address 127.0.0.1
Port 8001
End
BackEnd
Address 127.0.0.1
Port 8002
End
.. etc ..
BackEnd
Address 127.0.0.1
Port 8011
End
Session
Type BASIC
TTL 300
End
End
The Test Applications
Now onto the nature of the tests. So basically I didn’t want to get into anything too fancy, so I’m testing these things:
- Read operations
- Create operations
- Queries
- Update operations
- View rendering vs. writing directly to the response
I am not going to be testing any really complex operations as the more complex an application is the more it becomes application performance and not framework performance that is being compared. Moving on then I set-up the test with the scaffolding capability of each framework. So with Grails I created a domain class like:
class Book {
String title
String author
String description
Date dateCreated = new Date()
static constraints = {
title(nullable:false, blank:false)
author(nullable:false, blank:false)
}
}
Then ran the following command to generate the base controllers and views:
grails generate-all
Similarly with Rails I created the following table in mysql:
To create this yourself you can do it in MySQL’s tools or by running the DDL:
CREATE TABLE `books`.`books` (
`id` INT NOT NULL AUTO_INCREMENT,
`title` VARCHAR( 255 ) NOT NULL,
`author` VARCHAR( 255 ) NOT NULL,
`description` TEXT,
`date_created` TIMESTAMP NOT NULL,
PRIMARY KEY (`id`)
)
CHARACTER SET utf8;
I then created the following model in Rails:
class Book < ActiveRecord::Base
end
And then ran the command
ruby script/generate scaffold book
The Test Data
With that done, the way I setup some test data was to create an action to do so in each respective controller. In Grails I did this:
def data = {
def i = 1000
while( i > 0 ) {
def book = new Book()
book.title = “Book ${i}”
book.author = “Author ${i}”
book.description = “The description for book ${i}”
book.save()
i = i - 1
}
render “Created data!”
}
In Rails to achieve the same thing I did this:
def data
i = 1000
while i > 0
@book = Book.new
@book.title = “Book #{i}”
@book.author = “Author #{i}”
@book.description = “The description for book #{i}”
@book.save
i = i - 1
end
render :text => “Created data!”
end
Sending a request to each action created the data. Job done. Now we have some test data and the respective list views of each app look like this:
If you want to download the full test applications they can be downloaded from here:
Test Tools
To perform the tests I used ApacheBench which is available with the “ab” command on Mac OS X. Just for clarity here is the version info
This is ApacheBench, Version 1.3d <$Revision: 1.73 $> apache-1.3
Copyright (c) 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright (c) 1998-2002 The Apache Software Foundation, http://www.apache.org/
The results have then been graphed up using Apple Pages. The full PDF can be download here
Test A: Read Performance
So after creating the test data we now have 1000 records in each applications respective database. Just to be clear the code that renders the above view is the standard scaffolded list view for both apps with the server side code in Grails being:
def list = {
if(!params.max)params.max = 10
[ bookList: Book.list( params ) ]
}
And in Rails:
def list
@book_pages, @books = paginate :books, :per_page => 10
end
To run the read performance test I executed the commands:
ab -c 50 -n 1000 -e “rails_list_test.csv” http:
ab -c 50 -n 1000 -e “grails_list_test.csv” http:
Obviously not at the same time!
So what this does is send 1000 requests to the respective servers with 50 concurrency. I made sure all other applications were closed to make sure the figures were as accurate as possible. The results of the read test are below:
Grails is able to handle 40 requests per second to Rails 32. Significantly you can notice Grails has a much poorer worst case response time, but a superior mean response time (The mean time is the average time taken per request over all 1000 requests). This can be put down the Java threading model and the Thread pool in Tomcat at some point running out of available threads and blocking for a moment. It just reflects the differences in architecture more than anything else. If you ran the same test against Grails with a concurrency of 1 you would see linear performance as expected:
ab -c 1 -n 1000 -e “grails_list_test.csv” http:
Interestingly, Grails with only 10 threads is even more performant and is able to cope with 54 requests per second. We also see that the spikes in the max response time are gone with a configuration for less threads. If you think about it this makes sense given the hardware I’m testing on. I only have dual cores on this Macbook, so the more threads there are the more processes have to share the two cores and hence the overall performance degrades with more threads. If I was running a machine with 8 processors then having more threads would obviously be more beneficial.
Rails on the other hand with 10 mongrels and Pound degrades in performance. I guess this is down to more processes having to share the two cores, but if someone can explain if I’ve done anything wrong in the config please say so
Test B: Creating Records
The next test was to test creating new records. In this case I used a simple action for both. The Rails code is:
def createTest
i = 5
while i > 0
@book = Book.new
@book.title = “Book #{i}”
@book.author = “Author #{i}”
@book.description = “The description for book #{i}”
@book.save
i = i - 1
end
render :text => “Created 5 records data!”
end
As you can see it just creates 5 records in a loop and then renders some text to the response. No view involved. The Grails code for this is:
def createTest = {
def i = 5
while( i > 0 ) {
def book = new Book()
book.title = “Book ${i}”
book.author = “Author ${i}”
book.description = “The description for book ${i}”
book.save()
i = i - 1
}
render “Created 5 records data!”
}
To run the create performance test I executed the following ab commands framework:
ab -c 50 -n 1000 -e “rails_create_test.csv” http:
ab -c 50 -n 1000 -e “grails_create_test.csv” http:
Again with a 1000 requests, and 50 concurrency. The results are as follows:
The differences here are quite astounding, Grails is able to cope with 140 requests per second and has a mean average of 345ms per request compared to Rails’ 35 (and a half) and 1.3 seconds per request. I believe, without having done any real profiling, the reasons for this can largely put down to Hibernate’s session-based model. Each call to save() in ActiveRecord immediately persists the data whilst Hibernate batches updates up until the end.
Again Grails with only 10 threads is faster, whilst Rails with 10 mongrels degrades performance.
Test C: Querying
The next test tackled querying using a like query. The bottleneck here is likely to be MySQL’s query performance. The code for this test for Rails is as follows:
def queryTest
@books = Book.find(:all, :conditions => [“title like ?”, “Book 6%”])
end
The view used is the same as the default list view just cloned and renamed to queryTest.rhtml. For Grails the same thing is done:
def queryTest = {
[bookList:Book.findAllByTitleLike(“Book 6%”)]
}
There is no equivalent to Rails’ :conditions element in Grails so we use a finder expression. Again to execute this test we use similar commands:
ab -c 50 -n 1000 -e “rails_query_test.csv” http:
ab -c 50 -n 1000 -e “grails_query_test.csv” http:
The results for this test can be seen below:
As you can see the results here are much more equal as it is really reliant on database performance. Notably we can see Grails’ worst case is poor down to blocking in the thread pool, but overall the mean time of 5.387ms is superior to Rails’ 6.386, which illustrates why Grails’ is able to deal with 8.92 requests per second as oppose to Rails’ 7.65.
Neither framework comes off particularly well here, but in reality if you had this kind of load for queries you would need consider using some sort of caching and/or indexing solution (lucene/ferret).
This is the one test though, where Rails with 10 mongrels gets close to matching the default Grails tomcat configuration and where Grails with 10 threads is slower than the default tomcat config.
Test D: Updating record and rendering a view
For this test we’re trying out updating and then rendering a view. To do so we’re going to retrieve some random records from the database and update each and then store the value. To achieve this in Rails we do:
def updateTest
count = Book.count
ids = [rand(count),rand(count),rand(count)]
@books = []
ids.each { |id|
book = Book.find(id)
if book
book.author = “Fred Flintstone #{Time.new}”
book.save
@books << book
end
}
end
To explain this logic we basically create 3 random ids from the total, retrieve them from the database, update the author field and then save. And before you ask I definitely checked that the databases of both were dropped and re-created with fresh test data with the ids on the tables indexing from 1-1000. The view for this action is again copied from the list view and renamed to updateTest.rhtml.
For Grails to achieve the same thing we do:
static final rand = new Random()
def updateTest = {
def count = Book.count()
def ids = [rand.nextInt(count), rand.nextInt(count),rand.nextInt(count)]
def result = []
ids.each { id ->
def book = Book.get(id.toLong())
if(book) {
book.author = “Fred Flintstone ${new Date()}”
book.save()
result << book
}
}
[bookList:result]
}
Again the same logic, instead of the C rand(num) function in Ruby we use Java’s java.util.Random class and calculate a bunch of ids, iterate over each, saving each one to formulate the results.
Again to execute this test we use similar commands:
ab -c 50 -n 1000 -e “rails_update_test.csv” http:
ab -c 50 -n 1000 -e “grails_update_test.csv” http:
The outcome of this test can be seen below:
Once again we see a similar graph, Grails having the higher max, but a superior mean average response time. Grails is able to deal with 44.95 requests per second to Rails’ 30.88. This can be put down to a number of things, again I believe it is the difference between Hibernate’s session based approach and ActiveRecord.
Another thing to note is during the Grails test 2 Hibernate StaleObjectExceptions were thrown because Grails uses Hibernate’s optimistic locking feature, which ActiveRecord doesn’t seem to have any concept of (correct me if I’m wrong). In a real circumstance when dealing with high concurrency you would of course catch this exception in Grails, which would actually be a Spring HibernateOptimisticLockingFailureException and deal with it appropriately by getting fresh data or returning an error (the default case).
Again similar to the first two tests, we see Grails with only 10 threads being even more performant and Rails with 10 mongrels degrading.
Test E: Updating 3 random records and return idiomatic XML
So after the above result I wanted to see what the performance would be like when not using a view (RHTML or GSP). So instead we return some XML. In rails the code for this is:
def updateTest2
count = Book.count
ids = [rand(count),rand(count),rand(count)]
@books = []
ids.each { |id|
book = Book.find(id)
if book
book.author = “Fred Flintstone #{Time.new}”
book.save
@books << book
end
}
render_text @books.to_xml
end
Note here we call to to_xml method to auto-convert to XML which is a nice feature. The resulting XML looks something like:
<?xml version=“1.0″ encoding=“UTF-8″?>
<books>
<book>
<author>Fred Flintstone Thu Mar 22 15:35:54 +0000 2007</author>
<date-created type=“timestamp”>Thu Mar 22 12:20:21 +0000 2007</date-created>
<description>The description for book 3</description>
<id type=“integer”>5908</id>
<title>Book 3</title>
</book>
…
</books>
To do the same thing in Grails requires a little more code as Grails doesn’t have a toXML() yet (actually it does, but it is currently in the REST plug-in, see http://grails.org/Plugins). So instead of toXML() we use Groovy mark-up building:
def updateTest2 = {
def count = Book.count()
def ids = [rand.nextInt(count), rand.nextInt(count),rand.nextInt(count)]
def result = []
render(contentType:“text/xml”) {
books {
for( ident in ids ) {
def b = Book.get(ident.toLong())
if(b) {
b.author = “Fred Flintstone ${new Date()}”
b.save()
book {
author(b.author)
‘date-created’(b.dateCreated)
‘description’(b.description)
id(b.id)
title(b.title)
}
}
}
}
}
}
The result of the two methods is exactly the same XML (changing data of course). Now for running the tests, again the same commands:
ab -c 50 -n 1000 -e “rails_update_test2.csv” http:
ab -c 50 -n 1000 -e “grails_update_test2.csv” http:
And the result is:
So that is pretty similar to the first update test. Again Grails is around 40-50% faster. Now there is a lot of dynamic stuff going on when producing the XML from these two actions so I thought I would have a go at building up a String and writing it to the response.
Again similar to the first two tests, we see Grails with only 10 threads being even more performant and Rails with 10 mongrels degrading.
Test F: Updating 3 random records and returning a String generated response
This test actually is more a like for like as we don’t have to deal with the differences between to_xml in Rails and mark-up building in Grails. The Rails code for this is:
def updateTest3
count = Book.count
ids = [rand(count),rand(count),rand(count)]
xml = “<books>”
ids.each { |id|
book = Book.find(id)
if book
xml << “<book>”
book.author = “Fred Flintstone #{Time.new}”
book.save
xml << “<author>#{book.author}</author>”
xml << “<date-created>#{book.date_created}</date-created>”
xml << “<description>#{book.description}</description>”
xml << “<id>#{book.id}</id>”
xml << “<title>#{book.title}</title>”
xml << “</book>”
end
}
xml << “</books>”
render_text xml
end
The XML that is produced is exactly the same as in the previous test, except constructed manually from a String. The equivalent Grails code looks like:
def updateTest3 = {
def count = Book.count()
def ids = [rand.nextInt(count), rand.nextInt(count),rand.nextInt(count)]
def xml = new StringBuffer(“<books>”)
ids.each { id ->
def b = Book.get(id.toLong())
if(b) {
b.author = “Fred Flintstone ${new Date()}”
b.save()
xml << “<book>”
xml << “<author>${b.author}</author>”
xml << “<date-created>${b.dateCreated}</date-created>”
xml << “<description>${b.description}</description>”
xml << “<id>${b.id}</id>”
xml << “<title>${b.title}</title>”
xml << “</book>”
}
}
xml << “</books>”
render xml.toString()
}
Again it uses the static java.util.Random instance and constructs the String from a StringBuffer. Now we execute the commands:
ab -c 50 -n 1000 -e “rails_update_test3.csv” http:
ab -c 50 -n 1000 -e “grails_update_test3.csv” http:
Again with a 1000 requests, and 50 concurrency. The results are as follows:
Grails in this case is able to process nearly 3 times as many requests as Rails per second. So why are the first 2 update tests so much slower for Grails (although still faster than Rails?). Well what these tests have helped us identify is that with a large amount of load Groovy Server Pages, Grails’ view technology, certainly needs some improvement.
And again similar to the first two tests, we see Grails with only 10 threads being even more performant and Rails with 10 mongrels degrading.
The way GSP works internally with tag libraries is that it uses try/catch blocks to intercept exceptions thrown by Groovy when it doesn’t find a method. These are of course expensive as the JVM has to take a snapshot of the exception when it is thrown which causes all threads to freeze momentarily. So one area where we can improve Grails’ performance is to look at tweaking GSP to not do this (there are other techniques for doing this).
So why does this effect the second example? Groovy’s streaming markup builder is using the same technique!
Summary
Overall, as expected, given its base on solid Java technologies, Grails out performs Rails (100% Ruby) in pretty much every test. As I said at the start I’m no Rails performance tuning wizard, so if someone can suggest ways to get Rails to perform better please say so. We are conducting these tests purely to inform concerned Grails users who are asking for figures, and hope to review them as Grails progresses.
Interestingly, Grails, at least on this machine, sees performance increases when lowering the thread pool to 10, whilst Rails suffers when 10 mongrels are configured with Pound.
Grails hasn’t undergone much optimisation at all yet and there are certainly areas we can improve on, GSP rendering being one. GSP is of course a much more complex view rendering technology in comparison to RHTML due to its support for dynamic tag libraries, plus Sitemesh decoration. However, there are definitely areas in GSP, such as the way it uses exceptions, that can be improved upon.
Another point to remember is performance != scalability. The way you would scale an application to deal with the kind of load we have here differs greatly between Grails & Rails I imagine. With Grails you could stick a Terracotta or Coherence distributed cache on the back of it to implement SSI (singe system image) or use HTTP session replication across a cluster whilst with Rails you could ??? (a Rails user please feel free to provide an answer to this question in the comments, I’m not qualified to answer).
Nevertheless, I am quite satisfied that Grails performs sufficiently well to be competitive and there is little point in optimising prematurely. Our main goals is to focus on getting 1.0 out towards the middle of this year.
Resources
– Grails.org
Tags: Groovy
and http://blog.shinetech.com/?p=18
About
