“On the night of July 9, 1958 an earthquake along the Fairweather Fault in the Alaska Panhandle loosened about 40 million cubic yards (30.6 million cubic meters) of rock high above the northeastern shore of Lituya Bay. This mass of rock plunged from an altitude of approximately 3000 feet (914 meters) down into the waters of Gilbert Inlet (see map below). The impact generated a local tsunami that crashed against the southwest shoreline of Gilbert Inlet. The wave hit with such power that it swept completely over the spur of land that separates Gilbert Inlet from the main body of Lituya Bay. The wave then continued down the entire length of Lituya Bay, over La Chaussee Spit and into the Gulf of Alaska. The force of the wave removed all trees and vegetation from elevations as high as 1720 feet (524 meters) above sea level. Millions of trees were uprooted and swept away by the wave. This is the highest wave that has ever been known.“ (quoted from http://geology.com/records/biggest-tsunami.shtml)

Now lets use our imagination a bit, and pretend we’re on a digital world, and that an even bigger wave can be seen on the horizon, only that the wave is made up of 1’s and 0’s. That’s the current status of information on the net right now.

A huge wave of data is being generated every second, ranging from user generated information such as tweets, status updates, uploaded pictures, blog posts, comments, text messages, e-mails and so on to machine generated data, like server access logs, error logs, transaction logs, etc.

And that’s not even the problem, the problem is that we need to start thinking in terms of TB or even PB of information, billions of rows instead of millions of them in order to be able to handle this big wave that’s coming.

Normally, when you have to store information on your application you ask yourself one basic question:

What do I need this information for?

And from the answer you get, you plan your storage and you start saving that specific information.

Lets look at an example, from two different perspectives:

Traditional way of thinking:

Say for example, you’re a web development company and you’re asked to create a basic web analytics  app for your company site.  So you ask yourself:

What do I need the  information for?

As an answer, you might get something like:

• To get number of visits to each page.
• To get a list of referrer sites.
• To get the number of unique visits.
• To get a list of web browsers used on the site.

It’s a short list, I know, but this is a basic example.

Back to the problem:

You have your answer, all that information can be fetched from the server’s access log, so you configure your log files to store that information, great! You’re done!

Yes, you’re done, you got your system ready, it shows the information you were asked to show, but you also closed the door to other potential analytics that could come out of the information stored on those access logs (like request method used, response code given, size of the object returned and so on) and other sources of information.

Thinking in “big data” terms:

Thinking in “big data” terms means (at least to me), saving all the information you’re working with on your project and then finding out new and exiting ways to interpret that information and get results out of it.

Back to the problem, with the “big data” way of thinking this time:

This time around, you think in “big data” terms, so you already have lots of data being saved for every visit, such as:

• Access log information.
• Error log information.
• User input (if there is any)
• User behavior data (such as clicking patterns and smiliar)
• and so on.

That’s because when you created your website, you asked yourself a different question:

What is all the information I can get from my website?

And since you changed your question, you significantly change the answer to your problem.

You now have a vast amount of information to analyse and get insight from.

This is great, but where do we store all this log information? It could potentially become too much for a single machine and we don’t want to loose any information by rotating logs and using other techniques.

So another valid question would be:

What kind of hardware do I need to store and process all that information in a timely manner?

What kind of hardware do we need then?

We need some kind of setup that will allow us to:

• Store vasts amounts of data
• Process this data in a timely manner
• Be able to grow as much as we want (storage and processing power wise)
• Be fault tolerant (storage and processing power wise)
• Affordable

That is a lot to ask (specially if we consider the last point) of a single computer, isn’t it? So the answer will probably come in the form of a distributed system.

Enter Hadoop

What is Hadoop? In a nutshell, Hadoop is the solution to our problems, one of them (mind you), but a pretty powerful one at that.

In more detail, Hadoop is an Open Source Apache project, dedicated to solve two major problems related to big data:

1. Where to store all of the information?
2. How to process that information at an affordable cost and in an reasonable amount of time?

To answer these questions, Hadoop provides the following solutions:

HDFS

This is the Hadoop Distributed File System, it allows us to store in a reliable way all the information we need.
This works by interconnecting commodity machines (affordable) and using the resulting shared storage (Store vasts amounts of data).
The HDFS takes whatever we throw at it and splits the files into evenly sized chunks of data, and then spreads them throughout the cluster. In this stage, it also replicates the files, providing data redundancy and fault tolerance.

Thanks to the HDFS we can have  as much storage capacity as we need, by adding new machines to the cluster (Be able to grow as much as we want ).
We also gain a very important asset, that is fault tolerance. Since we’re replicating  the information into several nodes of the cluster, our commodity machines are free to fail and the only place where that will affect us is performance (no data loss or incomplete information).

MapReduce

This is the other “leg” of Hadoop, an implementation of the MapReduce algorithm proposed by Google in 2004.

The MapReduce algorithm allows us to process large amounts of information (terabytes of information) in a distributed (thouthend of nodes) and fault tolerant manner (Process this data in a timely manner) . And if we consider that we already have a cluster of computers working for us with the HDFS, MapReduce is the perfect match to take advantage of that computational power sitting there on every node of the cluster.

This algorithm has two basic steps:

1. Map step: In this stage, the input data will be split into smaller chunks to be analyzed and transformed by processes called “mappers”. Thanks to the integration with the HDFS, the main node will effectively schedule map jobs to use the data that’s already on the nodes they’re running on, allowing the system to utilize very little bandwidth. The output of these mapper jobs will be a set of (key, value) tuples.
2. Reduce step: The output of the mappers will be sent into the reduce jobs. These jobs will process the information with an added benefit of knowing that it’s input will be given in a sorted manner by the system. They’re main purpose is to aggregate the information given by the mappers and output only that which is needed.

There is an implicit step between 1 and 2, that is the shuffle & sort step, done by the system automatically. In this step, the system will sort the output of all mapper nodes by the key of each tuple and it’ll send these sorted results into the reduce nodes, assuring that all tuples with the same key will go to the same reducer.

Thinking the Hadoop way

So, we have our data, we have our questions to ask to that data, we have our needs and we have our solution. What now?

Your following steps could include:

1. Installing and configuring your Hadoop cluster:  For this step, the company Cloudera has a standarized distribution of hadoop, which they call Cloudera’s Distribution Including Apache Hadoop(CDH). You can download it for free and it comes with serveral other projects from the Hadoop ecosystem (such as Pig, Hive, Hbase, and so on). And for managing your cluster, you could use their Cloudera Manager, which allows you to manage up to 50 nodes for free.
2. Upload the information to your HDFS.
3. Transform your information using a MapReduce job: I consider this step to be optional. I would use a “hand-written” MapReduce job if I had transform my data set in a specific way in order to query it later on.
4. Query your data set: There are several ways to do this, tools like Pig or Hive, allow you write MapReduce jobs (for data transformation) on a higher level language (PigLatin or SQL). Others like HBase and Cassandra work better for quick queries to that data, they work directly over the HDFS ignoring the MapReduce framework, but you’re a bit limited on what you can do with the information.

And finally, a pretty common question:

Is Hadoop the best solution for big data analysis out there?

Probably not, since “the best” is always relative to your needs, but it’s a pretty darn good one, so give it a try.

Besides, all the cool kids are doing it:

• Facebook – 15 PB of information last time they revealed the number.
• Ebay – 5.3 PB of information on their clusters.
• LinkedIn
• Twitter

And many others, check out the complete list here.

Kids Ruby (to kids)

http://kidsruby.com: Kids Ruby is especially focused on kids, with a very easy interface which allows you to see the code, run it, and at the same time see what it outputs. Kids Ruby is also attractive for his Turtle graphics that allows you to draw pictures and have fun by mixing and trying colors. Kids Ruby includes a lot of useful resources and you don’t even need an internet connection to work. Developers also created a complete KidsRuby operating system based on Ubuntu Linux that makes program in Ruby a lot easier for kids.

Rails for zombies

http://railsforzombies.org/: Rails for Zombies offers an open-source web framework with all the power of the Ruby language and with no additional configuration needed. In this site you can see tutorial videos which allow you to learn more about Ruby on Rails in just five levels. After seeing each video you’ll be challenged with cool exercises to practice your new skills. So if you’re a zombie and you’re hungry for Ruby’s knowledge this is the perfect site for you.

Try Ruby

http://tryruby.org/: This website brings a very interactive Ruby tutorial; you can test new functions step by step and understand a little bit more about this language. In just 15 minutes and with a very interactive interface you can understand what Ruby is about. This site also allows you to save your progress by sign up for free at Code School.

Now that you have these very easy options to learn Ruby why don’t you try it out and maybe we’ll see you soon as a new member of Moove-IT’s team

Stop Online Piracy Act (SOPA) and Protect IP Act (PIPA) will be voted on Jan 24^th by the Congress in attempt to pass internet censorship in the Senate. These two laws are probably the most rejected ones by Americans citizen because some of them consider that they affect the most appreciated thing on internet, freedom.

The SOPA law attempt to close any foreign site which sells or shares pirated content from the USA, including music, films, books and every product non authorized for free distribution on the internet. The PIPA law meanwhile has its focus directly on protect Intellectual Property Act, avoiding any economic threats and thefts to creativity.

Anyway, these laws have great support from big industries like National Cable & Telecommunications Association, the National Association of Theatre Owners, Viacom, Copyright Alliance and NBC Universal, which argue that their businesses are dramatically affected by online piracy.

We will have to wait until next January 24 to see if the public opinion will have a direct influence on the fate of these laws.

Wikipedia.org – Home page Jan 18^th

Google.com – Home page Jan 18^th

WordPress.com – Home page Jan 18^th

Here at moove-it, we always put ourselves in our clients shoes to understand their needs and give them exactly what they are looking for. In order to do so we’ve created the UX Department.

From the moment we started gathering up to discuss latest design patterns and the top UX tendencies, we knew something great would come out of it, and so it did. We managed to nurture the whole team with what we’ve learned, improve existing products and enhance new projects’s design from scratch.
We’ll keep working as hard as possible on UX, not only because of how thrilled we’ve got with the results, but also because the way the users feel the product is what matters most.

We present partial conclusions found by the UX group. We share the presentation (in spanish)

There was a problem in one of our projects at moove-it related to slow processing. There is a daemon spawning new threads to process certain new entries to the database. The entries come from a different system, that’s the reason for this program which processes each new entry. Sometimes at certain hours of a day there are peaks in the entries to the database and the process will fall behind by about 20.000 entries or more. So we started analyzing the code to understand what was happening and why it took so long. We noted that each new thread the daemon spawned took about 5 seconds to complete its task. As we narrowed the measurement we came up with some code that took 5 seconds to execute but it only involved access to the database. So thanks to Aaron Patterson’s (@tenderlove) talk at RubyConf Uruguay about “Who makes the best asado” where he talked about rails and how it manages threads and database connections, we knew where to look.

What he explained is that each new thread requests its own database connection from the connection pool, and if there isn’t a free connection, rails waits for about 5 seconds and if after that there is no free connection it iterates over all the threads to take back the connections of dead threads(more info). See the correlation with the 5 seconds I talked about in the previous paragraph? We immediately suspected that this was the problem. So we started searching the Rails API for a way to release the connection at the end of each thread’s execution. Surprisingly, we didn’t find an easy and understandable explanation anywhere at first googling , so we digged deeper and came up with the following line:

ActiveRecord::Base.connection_handler.clear_active_connections!

The ActiveRecord::Base.connection_handler method returns the connection handler for the current thread and the clear_active_connections method does what it looks like, or from the Rails doc: “Returns any connections in use by the current thread back to the pool, and also returns connections to the pool cached by threads that are no longer alive.”

So this line returns the connections in use by a thread to the pool and enables the new threads spawned by the daemon to use the freed connections. This way we avoid the 5 second wait for rails to free the connections for us.

This one line picked up our performance from processing 1.000 entries in almost 2 hours to processing 10.000 in 5 minutes. Nice huh?!

That’s it. I’m not sure if this is the best way of doing it since this method also “… returns connections to the pool cached by threads that are no longer alive.” I guess this means it does the iteration over all the threads Aaron mentioned, but as you can see I’m happy with the performance improvement. We are using Rails 3.0.5, Aaron said that he will change the behavior, read more about it here.

Special thanks to @cheloeloelo who helped detecting the problems and digging through the Rails API finding the proper method to free the connections.

Image: Suat Eman / FreeDigitalPhotos.net

Nike 10K consist in running 10 kms along the coastline through some Montevideo’s neighborhoods enjoying the beautiful landscape.

Take a look at the pictures.

The whole team 

Silvana, Martin and Nicolas

Ariel before breaking the finish ribbon  :-P

Bird´s eye view

RubyConf Uruguay 2011

We are hungry for knowledge and refreshment, and we all want to be on the same train.

Please welcome all the new members to this awesome community. And help spreading the news, but even more important, do not miss the opportunity to meet the experts, discuss the future of RoR, and be Rail!

It doesn’t matter if you listen to the drums once a month, everyday, or once a year.  When it comes to local music, there is nothing more Uruguayan than the sound produced by rhythmically striking a drum, and especially when playing Candombe, a unique way of percussion, which will make your hair stand on its end.

Team building activities can range from treasure hunts to Safari trips, though this time, we have decided to do one which people could easily identify with, and which does not require sophisticated skills, but the desire to unwind, switch off and connect. Drumming workshops come first, the bonding is just a consequence.

Last week we had our first percussion workshop. Pablo Leites, an excellent musician and percussionist, also known as “Gancho” has been our instructor.  He has also been Martin Cabrera’s (Moove-IT cofounder) best friend for a long time.

Please have a look at the following pictures…

Why percussion

Why is percussion so important to us? Being a country of immigrants, Uruguay was formed by people from all over the world. And like it always happens, music has played a tremendous positive role in bringing people together and creating stronger, more significant bonds. We connect with our basic instincts, we forget about language barriers, cultural differences, rank, and we just let ourselves feel and relax.  Just listen to a few strikes and you will feel multicolor, ageless and energized.

You may have heard of Las Llamadas (The Callings), a popular annual event during Carnival here in Uruguay, which gathers thousands of people from all over the world. The drums are the main stars, and the African music roots brought by the people once made slaves in this country (and happily freed more than 150 years ago) are now our truly genuine and local music.

I personally love this rhythm, and even though I am not a music expert I will recognize its pace wherever I go. I am not sure if it is the adrenaline than runs through your body, or the inseparable link to human nature, but percussion makes your body shake, like toddlers instinctively struggling to move their bodies to the rhythm of the music.

Team Building !!

What do they have in common and what makes them different?

Even though it’s hard to come up with a comparison table between all four alternatives, mainly because I can’t claim to have personal experience with all of them, the Internet has a lot of information on the subject, so I went ahead and did a bit of research on the matter.

Another point of interest to consider is that though on the long run, all four solutions provide very similar services; they do it a bit differently, since they can be categorized into two places:

• Full text search servers: They provide a finished solution, ready for the developers to install and interact with. You don’t have to integrate them into your application; you only have to interact with them. In here we have Solr and Sphinx.
• Full text search APIs: They provide the functionalities needed by the developer, but at a lower level. You’ll need to integrate these APIs into your application, instead of just consuming it’s services through a standard interface (like what happens with the servers). In here, we have the Lucene Project and the Xapian project.

Taking all of this into account, we can now proceed into a more in-depth discussion about our options:

Full text search servers

Like I’ve discussed before, search servers provide an end solution, ready to be installed, to be tweaked into your needs and to be interfaced with.

The installing and tweaking process can be hard, depending on your specific needs, but the bright side is that all of it is done outside your application and your code. Once you’re done and the server is installed according to your needs, the only step left is interacting with it from your application using one of the methods provided by the server.

So, what do they have in common?

• They’ll both satisfy your needs regarding searching and indexing speed, since they do it very efficiently.
• They both have a long list of high-traffic sites using them, as we’ve seen above.
• Both offer commercial support, which is great if you’re planning on developing a commercial application to use them.
• Both offer client API bindings for several platforms/languages.
• Both can be distributed to increase speed and capacity.
• They both have great support for advanced querying. This allows them to use proximity search, relevance sorting and so on.

Some differences:

• Solr is an Apache Project built on top of the Lucene Project, allowing it to improve some of it’s features whenever the Lucene projects updates. Sphinx is an isolated project, which requires it to improve independently.
• Solr supports the use of wildcards for it’s searches, while the current release version of
  Sphinx (0.9) does not. The latest beta for Sphinx (2.0) though, appears to be supporting this feature.
• Solr comes with spell check for search terms out of the box, while Sphinx does not.
• Solr can parse rich text formats like Word documents, PDF files and so on; Sphinx does not provide this feature out of the box.
• Sphinx integrates more tightly with RDBMSs, especially MySQL since it was built with this functionality in mind.
• Sphinx can use the stopwords for a more relevant result ranking by default, whether Solr extracts them before doing the search.
• Sphinx supports SQL as a query language, when Solr forces you to learn it’s own language.
• Sphinx does not support multithreading on Windows machines, which slows down it’s performance on this OS.

So, which one is better?

As you might have guessed, there is straight answer to this question, since they both do a similar job, but have different strong and weak points. This allows them to be better than the other for some specific cases.

On an out-of-the-box basis, for a generic search engine implementation, my opinion (which should be taken lightly, since it’s only the opinion of a single developer) would be to implement it with Solr.

This resolution comes based on the features that Solr provides out of the box. Some of them, being quite important for a search engine, such as wildcard support, rich text format parsing, and so on.

On the other hand, if your needs are as specific as indexing content from a database, then Sphinx would be the way to go, since it appears to have integration with that kind of content natively, which would assure a higher performance over other solutions.

Full text search APIs

In this case, we’ll be comparing APIs, which could be thought as “tools” to add to the “tool box” provided by the programming language you’re using.

Our two contestants are the Lucene Project and the Xapian project. They both have quite a number of followers and applications that use them, so lets see what else they have in common:

• They’re both highly portable. Lucene is written in JAVA, and thus it works on any JAVA capable OS. Xapian on the other hand is written in C++ and has support for most of the operative systems on the market.
• They both support rich text formats, which is definitely a plus if you’re trying to index any type of information (no need to pre-process them yourself).
• They both support advance search mechanics, like wildcards, proximity search, stemming, and so on.
• They’re both able to add data into their index on a real-time basis, making information accessible immediately.

Basically, Lucene and Xapian are both very much alike: they’re both very powerful and very customizable. And yet, they have some differences, as we’ll see next.

Some of the differences

• Lucene does not support faceted search, whilst Xapian does.
• Lucene does not support spelling corrections out of the box, whilst Xapian does.
• Xapian has support for synonyms out of the box, Lucene does not.
• Lucene has been ported into other programming languages (like PHP, Ruby, C#, etc, etc) providing Lucene-like APIs for those languages.
• Xapian provides bindings for other languages, but maintains a core API that’s always the same, securing that no matter which binding it is that you’re using, you’re always working with the official distribution and not a poorly done copy.

So, which one is better?

In this case, the answer (according to my research) to this question is not, as one would expect, the same as the one between Sphinx and Solr.

According to what we can see from our list of common and different features, one would be inclined to believe that Xapian is the way to go (at least I am!) since it is clearly superior (on an out-of-the-box basis of course).

So, why would we ever pick Lucene? Well, for starters, Lucene’s ports tend to have a good integration with some of those languages’ most famous frameworks (on php, Zend framework implements the Lucene search API, Ruby has it’s own implementation as well, called Ferret, which in turn has a RoR plug-in, and so on), which would ease the development process considerably. This could actually be a major point in favor of Lucene, if what we’re already using (or are planning on using) one of those frameworks.

Final thoughts

All in all, there are many solutions out there worth the try, even though I’ve only covered those that would appear to be the four most known (or used) options. There are others like DataparkSearch, Ht-//Dig, mnoGoSearch, KinoSearch, and a very long etc, which might be the right pick for your needs.

What is really important to remember is that database engines are not the only solution out there for data handling. And also that full text search solutions are not the silver bullet needed to kill the proverbial werewolf that represents our searching problems either.

We need to think about our needs very carefully before choosing a technology or we can end up having that proverbial wolf biting our rear…