There was a problem in one of our projects at moove-it related to slow processing. There is a daemon spawning new threads to process certain new entries to the database. The entries come from a different system, that’s the reason for this program which processes each new entry. Sometimes at certain hours of a day there are peaks in the entries to the database and the process will fall behind by about 20.000 entries or more. So we started analyzing the code to understand what was happening and why it took so long. We noted that each new thread the daemon spawned took about 5 seconds to complete its task. As we narrowed the measurement we came up with some code that took 5 seconds to execute but it only involved access to the database. So thanks to Aaron Patterson’s (@tenderlove) talk at RubyConf Uruguay about “Who makes the best asado” where he talked about rails and how it manages threads and database connections, we knew where to look.

What he explained is that each new thread requests its own database connection from the connection pool, and if there isn’t a free connection, rails waits for about 5 seconds and if after that there is no free connection it iterates over all the threads to take back the connections of dead threads(more info). See the correlation with the 5 seconds I talked about in the previous paragraph? We immediately suspected that this was the problem. So we started searching the Rails API for a way to release the connection at the end of each thread’s execution. Surprisingly, we didn’t find an easy and understandable explanation anywhere at first googling , so we digged deeper and came up with the following line:

ActiveRecord::Base.connection_handler.clear_active_connections!

The ActiveRecord::Base.connection_handler method returns the connection handler for the current thread and the clear_active_connections method does what it looks like, or from the Rails doc: “Returns any connections in use by the current thread back to the pool, and also returns connections to the pool cached by threads that are no longer alive.”

So this line returns the connections in use by a thread to the pool and enables the new threads spawned by the daemon to use the freed connections. This way we avoid the 5 second wait for rails to free the connections for us.

This one line picked up our performance from processing 1.000 entries in almost 2 hours to processing 10.000 in 5 minutes. Nice huh?!

That’s it. I’m not sure if this is the best way of doing it since this method also “… returns connections to the pool cached by threads that are no longer alive.” I guess this means it does the iteration over all the threads Aaron mentioned, but as you can see I’m happy with the performance improvement. We are using Rails 3.0.5, Aaron said that he will change the behavior, read more about it here.

Special thanks to @cheloeloelo who helped detecting the problems and digging through the Rails API finding the proper method to free the connections.

Image: Suat Eman / FreeDigitalPhotos.net

RubyConf Uruguay 2011

We are hungry for knowledge and refreshment, and we all want to be on the same train.

Please welcome all the new members to this awesome community. And help spreading the news, but even more important, do not miss the opportunity to meet the experts, discuss the future of RoR, and be Rail!

We share the presentations to everybody

All_in_One_Agile_Development (in spanish)

Ruby_and_Rails (in spanish)

Even when Ruby seems to be the most popular modern programming language in America, there was nowhere in Uruguay you could actually learn it. Moove-IT will give you a heads up on this exciting language and Rails framework on July 6, 2011 (please see ad below).

This will be the first in a series of workshops focusing on Ruby, RoR, Scrum and hands on training exercises.

We are so proud to have made so much progress working with this technology and methodology that we feel the need to contribute to the local Ruby community.

Join us! Be part.

First of all, we classified the available options into two different categories:

• ‘Native’ Development frameworks: Rhodes, NimbleKit, Titanium (this is new to the list), PhoneGap
• Javascript libraries: Sencha, JQTouch, JQuery mobile

Rhodes

According to the official description, The Rhodes application framework allows developers to create native mobile applications with portability of editing HTML templates and the power of the Ruby programming language. Applications written in Rhodes exhibit the performance and richness of apps written to the native device operating systems with local data but enable developers to have the productivity of web interfaces in HTML. Developers write their applications once and they then run on all major mobile device operating systems: iPhone, Windows Mobile, Blackberry and more.

Advantages:

• Programming in Ruby
• Full access to the Device API (Application looks different in each device)
• Several devices supported: IPhone, BlackBerry, Android, Windows mobile
• RhoSync: RhoSync retrieves data via web services (REST or SOAP) from backend enterprise applications for distribution to downstream mobile devices.

Disadvantages:

• Licensing: It is a commercial product. Costs arround U$S 500 per project.

NimbleKit

NimbleKit says that it is a framework to create applications for iPhone and iPod touch fast. Programmers do not need to know Objective-C or iPhone SDK. All they need is to know how to write an HTML page with Javascript code.

Advantages:

• HTML, CSS & Javascript

Disadvantages

• It only works for iphone, ipods & ipads.

Titanium & PhoneGap

From architectural standpoint, these two frameworks are very similar. Titanium and PhoneGap expose the smartphone features through a set of Javascript APIs, while the application’s logic (html, css, javascript) runs inside a native WebView control. Through the javascript APIs, the “web app” has access to the mobile phone functions such as Geolocation, Accelerometer Camera, Contacts, Database, File system, and so on.

The differences:

PhoneGap does not expose the native UI components to javascript. Titanium, on the other hand, has a comprehensive UI API that can be called in javascript to create and control all kinds of native UI controls. Utilizing these UI APIs, a Titanium app can look more “native” than a PhoneGap app. Second, PhoneGap supports more mobile phone platforms than Titanium does. PhoneGap APIs are more generic and can be used on different platforms such as iPhone, Android and Blackberry. Titanium is primarily targeting iPhone and Android. Some of its APIs are platform specific (like the iPhone UI APIs). The use of these APIs will reduce the cross-platform capability of your application.
If your concern for your app is to make it more “native” looking, Titanium is a better choice. If you want to be able to “port” your app to another platform more easily, PhoneGap will be better.

Advantages:

• Multi device
• Programming in HTML, CSS & Javascript
• MIT License

Disadvantages:

• Some devices native features are not exposed in the API

Sencha

According to the official documentation, Sencha Touch is the first app framework built specifically to leverage HTML5, CSS3, and Javascript for the highest level of power, flexibility, and optimization. They make specific use of HTML5 to deliver components like audio and video, as well as a localStorage proxy for saving data offline. They have made extensive use of CSS3 in our stylesheets to provide the most robust styling layer possible.
Altogether, the entire library is under 80kb (gzipped and minified), and it is trivial to make that number even smaller by disabling unused components or styles.

Advantages

• Great UI Controls
• Multi device

Disadvantages

• Commercial license (U$S 1.800 for 5 developers)

JQTouch
It is “A jQuery plugin for mobile web development on the iPhone, iPod Touch, and other forward-thinking devices.” [1]

Advantages

• Great UI Controls
• Multi device
• MIT License

Disadvantages

• Is focused on small screen devices
• It’s a jquery plugin
• It was created in 2009 by David Kaneda and has been relatively quiet until release of beta 3 (April 24th 2011).

JQuery Mobile
It is a unified user interface system across all popular mobile device platforms, built on jQuery and jQuery UI foundation. Its lightweight code is built with progressive enhancement, and has a flexible, easily themeable design.

Advantages

• Multi device (See http://jquerymobile.com/gbs/) (supports much more devices than JQTouch)
• Very simple to use
• John Resig, JQuery’s founder is one of the main developers and it is a high activity project
• It is not a jquery plugin, it is a new framework
• Dual licensed under MIT and GPL Version 2

Disadvantages

• It is still in Alpha version

CONCLUSION
After evaluating the different frameworks, we arrived at the next conclusions:
We should use a framework that generates html + css + js, because this technology guarantees a long term support on different devices
We should use a framework that supports access to devices specific capabilities (i.e. sound, gps, camera, vibration, data, etc.) in case we need them in a near future
Because we prefer unrestrictive licenses (MIT, BSD, LGPL and others) our options are reduced to: PhoneGap, Titanium & JQTouch.

Finally, we choose PhoneGap + JQuery mobile because they are open technologies and support multiple platforms. Also, resulting interfaces are similar across different devices and that is a desired feature for our users.
JQTouch and JQuery mobile are awesome frameworks to develop mobile applications, but we prefer to choose JQuery Mobile since it support much more devices (altough it is in alpha version), has high activity and is not just optimized for WebKit as jQtouch does.
We also think that JQuery Mobile is pretty easy to use, the resulting interfaces are usable and has many components to use.

The rest of the story is pretty easy ( just because we use Ruby on Rails ), you should create some helpers that adds the JQuery Mobile features to the pages, or use some existent plugin, you can find a starting guide here: http://goo.gl/5Soyq.

I hope you find this post useful, have a nice coding!

[1]: forward-thinking: Someone who is forward-thinking is thinking progressively and possesses the ability to look beyond the “now” and formulate strategies for future success. They are constantly asking “what’s next?”.

Sources:

• http://www.phonegap.com/
• http://www.nimblekit.com/
• http://rhomobile.com/
• http://www.jqtouch.com/
• http://jquerymobile.com/
• http://www.sencha.com/products/touch/
• http://fuelyourcoding.com/getting-started-with-jquery-mobile-rails-3/

_sprites.scss

@mixin sprite_for($image_class, $x, $y, $image: "/images/sprite_image.png") {
  .sprite_image.#{$image_class} {
    background: transparent url($image) no-repeat scroll $x $y;
    height:16px;
    width:16px;
  }
}
_images.scss

$my_sprite_image_path: "/images/my_sprite_image_path.png";

default.scss

@import "sprites", "images";

@include sprite_for("my_image", 0, 0, $my_sprite_image_path);

That generates

default.css

  .sprite_image.my_image {
    background: transparent url(/images/my_sprite_image_path.png) no-repeat scroll 0 0;
    height:16px;
    width:16px;
  }

So, this is best sorted using this method, because you have the image path in a single file and you use it in other files only by importing the file with the images path, it’s also simpler to remove an image because you only need to find the references to that variable and delete them.

The sass framework also allows you to use the DRY principle because you can write nested selector, I’ll show you with an example:

default.scss

.some_class {
  width: 100px;
  .some_other_class {
    display: block;
  }
}

default.css

.some_class {width:100px;}
.some_class .some_other_class {display:block;}

Doesn’t affect the performance because it generates a css file at development time.

It also allows you to benefit from some of the advantages of css 3 right now.

User.find(:all, : order => "created_at desc", :limit => 5)

where, having, select, group, order, limit, offset, joins, includes, ...

User.order("created_at desc").limit(5)

User.find(:all, :conditions => ["created_at <= ?", Time.now], :include => :tasks)

User.where("created_at <= ?", Time.now).includes(:tasks)

Named Scopes

class User < ActiveRecord::Base
named_scope :active, :conditions => '!is_deleted'
named_scope :south_american,
:conditions => ["nationality IN (?)", Nationality::south_american_countries]
end

class User < ActiveRecord::Base
scope :active, where('!is_deleted')
scope :south_american,
where("nationality IN (?)", Nationality::south_american_countries)
end

scope :recent, active.south_american.order("created_at desc")

validates_presence_of :email
validates_uniqueness_of :email
validates_length_of :email, :maximum => 30

validates :email, :presence => true,
:uniqueness => true,
:length => {:maximum => 30}

You are probably familiar with the N+1 query and cache counter problems, so let’s look how Bullet will detect these by monitoring the database queries.

The N+1 query problem

Suppose we have our application with accounts having many rate plans:

class RatePlan < ActiveRecord::Base
  belongs_to :account
end

If we iterate over the rate plans, accessing rate plans attributes, like:

RatePlan.all.each {
  |post| puts "#{rate_plan.name}, by #{rate_plan.account.name}"
}

Will be using N+1 queries:

• one query to get all the rate plans
• then, for each rate plan, one query to get the corresponding account.

(This means that if we have 1000 posts, we’ll be doing 1000+1 queries)

By using the :include option, we can retrieve the rate plans and their corresponding accounts on the same query:

RatePlan.find(:all, :include => :account).each {
   |post| puts "#{rate_plan.name}, by #{rate_plan.account.name}"
}

By detecting if we forgot to use eager loading or a counter cache, Bullet can save time looking for problems on the fly.
For example, Bullet will show this message:

We have to be careful not to abuse the :include option. If the rate_plans table is big enough, it could take too much server memory. Also, we could end up with big, slow requests, or getting from the server a lot of data that won’t be used. Bullet will show a message if it detects an unused eager loading.

Cache counters

Cache counters is another improvement in a one-to-many relationship. Because the _has_many_ relationship defines an attribute that is a collection, if we use the size property on this attribute, it will trigger a select count(*) on the child table. This is generally acceptable, except when we are using frequently count and we end up going to the database unnecessarily.
In order to avoid that, we can use counter caching: Active Record will maintain for a parent table the number of child references. Bullet also shows a message if it finds a cache counter is needed.

Configuration

You have to first install the gem, by adding it to the Gemfile, or by installing manually:

sudo gem install bullet --pre

The next thing is to add some configuration to the development environment. Bullet won’t do anything unless you configure it explicitly, and also note that it’s not a good idea to configure it on the production environment, because on that case your users would also receive those N+1 query alerts.

Configuration allows Growl notifications, and sending messages by XMPP (Jabber), but since the default JavaScript alerts are good enough for me, I configured it like this:

config.after_initialize do
  Bullet.enable = true
  Bullet.alert = true
  Bullet.bullet_logger = true
  Bullet.console = true
  Bullet.growl = false
  Bullet.rails_logger = true
  Bullet.disable_browser_cache = true
end

Note the last option, because disabling browser cache can save you some trouble on some configurations. In any case, if Bullet is not working, try first disabling browser cache.

With this configuration, it will show a JavaScript alert, the same message on the console, and the detail and stack trace on the bullet.log.

To do that task, you must add the following lines to the apache configuration.


<DirectoryMatch "^/.*/\.svn/">
  ErrorDocument 403 /404.html
  Order allow,deny
  Deny from all
  Satisfy All
</DirectoryMatch>