Here at moove-it, we always put ourselves in our clients shoes to understand their needs and give them exactly what they are looking for. In order to do so we’ve created the UX Department.

From the moment we started gathering up to discuss latest design patterns and the top UX tendencies, we knew something great would come out of it, and so it did. We managed to nurture the whole team with what we’ve learned, improve existing products and enhance new projects’s design from scratch.
We’ll keep working as hard as possible on UX, not only because of how thrilled we’ve got with the results, but also because the way the users feel the product is what matters most.

We present partial conclusions found by the UX group. We share the presentation (in spanish)

Nike 10K consist in running 10 kms along the coastline through some Montevideo’s neighborhoods enjoying the beautiful landscape.

Take a look at the pictures.

The whole team 

Silvana, Martin and Nicolas

Ariel before breaking the finish ribbon  :-P

Bird´s eye view

RubyConf Uruguay 2011

We are hungry for knowledge and refreshment, and we all want to be on the same train.

Please welcome all the new members to this awesome community. And help spreading the news, but even more important, do not miss the opportunity to meet the experts, discuss the future of RoR, and be Rail!

It doesn’t matter if you listen to the drums once a month, everyday, or once a year.  When it comes to local music, there is nothing more Uruguayan than the sound produced by rhythmically striking a drum, and especially when playing Candombe, a unique way of percussion, which will make your hair stand on its end.

Team building activities can range from treasure hunts to Safari trips, though this time, we have decided to do one which people could easily identify with, and which does not require sophisticated skills, but the desire to unwind, switch off and connect. Drumming workshops come first, the bonding is just a consequence.

Last week we had our first percussion workshop. Pablo Leites, an excellent musician and percussionist, also known as “Gancho” has been our instructor.  He has also been Martin Cabrera’s (Moove-IT cofounder) best friend for a long time.

Please have a look at the following pictures…

Why percussion

Why is percussion so important to us? Being a country of immigrants, Uruguay was formed by people from all over the world. And like it always happens, music has played a tremendous positive role in bringing people together and creating stronger, more significant bonds. We connect with our basic instincts, we forget about language barriers, cultural differences, rank, and we just let ourselves feel and relax.  Just listen to a few strikes and you will feel multicolor, ageless and energized.

You may have heard of Las Llamadas (The Callings), a popular annual event during Carnival here in Uruguay, which gathers thousands of people from all over the world. The drums are the main stars, and the African music roots brought by the people once made slaves in this country (and happily freed more than 150 years ago) are now our truly genuine and local music.

I personally love this rhythm, and even though I am not a music expert I will recognize its pace wherever I go. I am not sure if it is the adrenaline than runs through your body, or the inseparable link to human nature, but percussion makes your body shake, like toddlers instinctively struggling to move their bodies to the rhythm of the music.

Team Building !!

What do they have in common and what makes them different?

Even though it’s hard to come up with a comparison table between all four alternatives, mainly because I can’t claim to have personal experience with all of them, the Internet has a lot of information on the subject, so I went ahead and did a bit of research on the matter.

Another point of interest to consider is that though on the long run, all four solutions provide very similar services; they do it a bit differently, since they can be categorized into two places:

• Full text search servers: They provide a finished solution, ready for the developers to install and interact with. You don’t have to integrate them into your application; you only have to interact with them. In here we have Solr and Sphinx.
• Full text search APIs: They provide the functionalities needed by the developer, but at a lower level. You’ll need to integrate these APIs into your application, instead of just consuming it’s services through a standard interface (like what happens with the servers). In here, we have the Lucene Project and the Xapian project.

Taking all of this into account, we can now proceed into a more in-depth discussion about our options:

Full text search servers

Like I’ve discussed before, search servers provide an end solution, ready to be installed, to be tweaked into your needs and to be interfaced with.

The installing and tweaking process can be hard, depending on your specific needs, but the bright side is that all of it is done outside your application and your code. Once you’re done and the server is installed according to your needs, the only step left is interacting with it from your application using one of the methods provided by the server.

So, what do they have in common?

• They’ll both satisfy your needs regarding searching and indexing speed, since they do it very efficiently.
• They both have a long list of high-traffic sites using them, as we’ve seen above.
• Both offer commercial support, which is great if you’re planning on developing a commercial application to use them.
• Both offer client API bindings for several platforms/languages.
• Both can be distributed to increase speed and capacity.
• They both have great support for advanced querying. This allows them to use proximity search, relevance sorting and so on.

Some differences:

• Solr is an Apache Project built on top of the Lucene Project, allowing it to improve some of it’s features whenever the Lucene projects updates. Sphinx is an isolated project, which requires it to improve independently.
• Solr supports the use of wildcards for it’s searches, while the current release version of
  Sphinx (0.9) does not. The latest beta for Sphinx (2.0) though, appears to be supporting this feature.
• Solr comes with spell check for search terms out of the box, while Sphinx does not.
• Solr can parse rich text formats like Word documents, PDF files and so on; Sphinx does not provide this feature out of the box.
• Sphinx integrates more tightly with RDBMSs, especially MySQL since it was built with this functionality in mind.
• Sphinx can use the stopwords for a more relevant result ranking by default, whether Solr extracts them before doing the search.
• Sphinx supports SQL as a query language, when Solr forces you to learn it’s own language.
• Sphinx does not support multithreading on Windows machines, which slows down it’s performance on this OS.

So, which one is better?

As you might have guessed, there is straight answer to this question, since they both do a similar job, but have different strong and weak points. This allows them to be better than the other for some specific cases.

On an out-of-the-box basis, for a generic search engine implementation, my opinion (which should be taken lightly, since it’s only the opinion of a single developer) would be to implement it with Solr.

This resolution comes based on the features that Solr provides out of the box. Some of them, being quite important for a search engine, such as wildcard support, rich text format parsing, and so on.

On the other hand, if your needs are as specific as indexing content from a database, then Sphinx would be the way to go, since it appears to have integration with that kind of content natively, which would assure a higher performance over other solutions.

Full text search APIs

In this case, we’ll be comparing APIs, which could be thought as “tools” to add to the “tool box” provided by the programming language you’re using.

Our two contestants are the Lucene Project and the Xapian project. They both have quite a number of followers and applications that use them, so lets see what else they have in common:

• They’re both highly portable. Lucene is written in JAVA, and thus it works on any JAVA capable OS. Xapian on the other hand is written in C++ and has support for most of the operative systems on the market.
• They both support rich text formats, which is definitely a plus if you’re trying to index any type of information (no need to pre-process them yourself).
• They both support advance search mechanics, like wildcards, proximity search, stemming, and so on.
• They’re both able to add data into their index on a real-time basis, making information accessible immediately.

Basically, Lucene and Xapian are both very much alike: they’re both very powerful and very customizable. And yet, they have some differences, as we’ll see next.

Some of the differences

• Lucene does not support faceted search, whilst Xapian does.
• Lucene does not support spelling corrections out of the box, whilst Xapian does.
• Xapian has support for synonyms out of the box, Lucene does not.
• Lucene has been ported into other programming languages (like PHP, Ruby, C#, etc, etc) providing Lucene-like APIs for those languages.
• Xapian provides bindings for other languages, but maintains a core API that’s always the same, securing that no matter which binding it is that you’re using, you’re always working with the official distribution and not a poorly done copy.

So, which one is better?

In this case, the answer (according to my research) to this question is not, as one would expect, the same as the one between Sphinx and Solr.

According to what we can see from our list of common and different features, one would be inclined to believe that Xapian is the way to go (at least I am!) since it is clearly superior (on an out-of-the-box basis of course).

So, why would we ever pick Lucene? Well, for starters, Lucene’s ports tend to have a good integration with some of those languages’ most famous frameworks (on php, Zend framework implements the Lucene search API, Ruby has it’s own implementation as well, called Ferret, which in turn has a RoR plug-in, and so on), which would ease the development process considerably. This could actually be a major point in favor of Lucene, if what we’re already using (or are planning on using) one of those frameworks.

Final thoughts

All in all, there are many solutions out there worth the try, even though I’ve only covered those that would appear to be the four most known (or used) options. There are others like DataparkSearch, Ht-//Dig, mnoGoSearch, KinoSearch, and a very long etc, which might be the right pick for your needs.

What is really important to remember is that database engines are not the only solution out there for data handling. And also that full text search solutions are not the silver bullet needed to kill the proverbial werewolf that represents our searching problems either.

We need to think about our needs very carefully before choosing a technology or we can end up having that proverbial wolf biting our rear…

Today I want to write about Sebastian Borrazas and the programming contest he is taking part in.

Sebastian Borrazas is a talented developer, member of an exclusive group of developers which makes Moove-IT a big company – I’m not talking about quantity, I’m talking about greatness -.

Last week, he together with his team, won the Programming Contest organized annually by the ORT University of Montevideo, Uruguay.

The contest was basically about Programming Algorithms and Algorithm Design Techniques. Each group of contestants had to apply techniques like Backtracking, Dynamic programming, Divide & Conquer, Knapsack, Dijkstra, etc.

The challenge was clear: to successfully do the greatest number of exercises in no more than 4 hours. Then the jury would evaluate the execution time of the Algorithm and have a winner.

Next step will be to travel to Buenos Aires, Argentina on November 5 to take part in the regionals. This will be hard. More than 500 teams from all over Latin America will be there too.

We are very proud to have Sebastian as our Geek Ambassador in this contest!

Good luck Moovetiano!

Depending on the application being developed, one of these solutions can be more than enough to meet our needs and that is what we usually end up using.

But there are times, when the amount of information to be handled is so big (we’re talking about millions of rows of information) and the needed response times are so low (we’re talking about a few milliseconds time) that we need solutions designed specifically for searching large amount of information instead of generic data handling ones.

What’s so good about them?

While most database engines work great with very dynamic loads of information (i.e.: where new information needs to be shown in real-time), and managing highly complex relational data, sometimes what we need is actually a solution that can search through a great amount of data in a very short time, giving the user a quick response.

The solutions we’ll be talking about here, do just that, they give the developers a way of accessing information at a striking speed, but at a price: the information used is not always the most updated one¹. Another lose to consider here, is the fact that these searching solutions do not manage relational information like relational databases do. So you need to be careful when choosing one over the other, and consider exactly what your needs are.

How do they work?

The way they work, is, roughly, by indexing all the information, using some of several indexing mechanics (like the use of stop words², stemming³, etc). Once the information has been indexed, it is there (on the index) where the search takes place, instead of searching directly on the unprocessed data.

Some of the most interesting features presented by these APIs are:

• Ranked search: The results can be returned ordered by the relevance of the match.
• Proximity search: Because sometimes, the searched terms are not written correctly, these APIs can return results that are close to the original ones.
• The use of wildcards: When searching, not always do we know the exact words, that’s when wildcards come into place, allowing us to specify a basic structure for our search and letting the search engine find the matching results.

Comparing some of them

Considering this scenario, today we’ll be discussing some alternative options for data handling and querying. I’ve chosen these projects based on general opinion on internet, according to most places, these guys are the most known/used alternatives out there (there are many others of course).

Our options are:

• Lucene: An Apache developed project, written entirely in Java.
• Xapian: An Open Source Search Engine Library, released under theGPL and written in C++.
• Sphinx: An open source full text search server with a high level of integration with database engines.
• Solr: Solr is a standalone enterprise search server with a REST-like API that uses Lucene as its base full-text search engine.

What am I looking for?

Before even comparing products, it is important to understand that there is rarely a single solution to our problems. That almost every time, there is more than one product that will meet our needs.

Knowing this, I’ll be comparing products based on an “out-of-the-box” basis. This means, that no addons or plugins or anything will be taken into account for comparison purposes (we have to draw the line somewhere).

My chosen criteria

I won’t be benchmarking product’s speed against each others, if that’s what you’re looking for then stop reading here. Even though it might be a good measure of how good a product is (in some cases) I feel that in this particular case, where we don’t really have a specific problem to solve, those numbers would not bring anything of value to the table.

What I will be taking into account is the amount of features (search related features actually) that come with a product out of the box. I fell that this approach could give a basic understanding of which product stands over others for any generic given problem. In the end, based solely on this information, my resolution can be wrong (or misleading) for some specific cases, but then again, that’s not what I’m trying to solve, since in order to tackle specific scenarios this article would have to be extended into a small book.

Another point of notice, is the fact that all of the presented solutions (as we’ll see in a minute), are long term projects, with active support and under constant development (and improvement).

The projects

Now that we know what we’ll be comparing and what to look for on our projects, lets take a look at them:

Lucene

Apache Lucene(TM) is a high-performance, full-featured text search engine library written entirely in Java. It is a technology suitable for nearly any application that requires full-text search, especially cross-platform.

It has support for indexing several industry standards, like Word documents, Excel, PowerPoint, PDF, HTML files and many more.

Basic features

High performance indexing

• Over 95GB/hour on modern hardware
• Incremental indexing as fast as batch indexing
• Index size roughly 20-30% the size of text indexed

Powerful, Accurate and Efficient Search Algorithms

• Ranked searching — best results are returned first
• Many powerful query types:
  • Phrase queries: allows for more than one word to be used on the search.
  • Use of wildcards: The API supports the use of “*” and “?” as wildcards on queries.
  • Proximity queries: Use of ~ to find look-a-like words.
  • Range queries: Used for range of numbers (1 to 10), sorted words (Amanda to Sabrina), date range and more.

• Fielded searching (e.g., title, author, contents)
• Sorting by any field
• Multiple-index searching with merged results
• Allows simultaneous update and searching

Cross-Platform Solution

• Available as Open Source software under the ApacheLicense which lets you use Lucene in both commercial and Open Source programs
• 100%-pure Java
• Implementations in other programming languages available that are index-compatible
  • PHP: ZendSearch
  • Ruby: Ferret
  • C++: CLucene
  • Many others!

Who uses it?

This is a widely used API, and some of the major software companies that use it are:

• Eclipse: The Eclipse IDE uses Lucene for searching it’s documentation.
• CNETReviews – Product Listing pages with Filter drill downs.
• IBMOmniFindPersonale-mailSearch – A semantic search engine for your e-mail.
• Others like Linkedin, AOL, etc (full list here)

Xapian

Xapian, like Lucene, is another open source, widely used full text search API, but unlike the first one, this one is written in C++.

Some basic features:

Cross-platform

• Highlyportable – runs on Linux, Mac OS X, FreeBSD, NetBSD, OpenBSD, Solaris, HP-UX, Tru64, IRIX, and probably other Unix platforms; as well as Microsoft Windows and OS/2.

• Written in C++.
  • Perl bindings are available in the moduleSearch::XapianonCPAN.
  • Java JNI bindings are included in the xapian-bindings module.
  • It supportsSWIG which generates bindings for many languages like Python, PHP, Tcl, C#, Ruby and Lua.

Great algorithms:

• Ranked probabilistic search – important words get more weight than unimportant words, so the most relevant documents are more likely to come near the top of the results list.
• Relevance feedback – given one or more documents, Xapian can suggest the most relevant index terms to expand a query, suggest related documents, categorize documents, etc.
• Phrase and proximity searching – users can search for words occurring in an exact phrase or within a specified number of words, either in a specified order, or in any order.
• Full range of structured boolean search operators (“stock NOT market”, etc). The results of the boolean search are ranked by the probabilistic weights. Boolean filters can also be applied to restrict a probabilistic search.
• Supports stemming of search terms (e.g. a search for “football” would match documents which mention “footballs” or “footballer”). This helps to find relevant documents which might otherwise be missed. Stemmers are currently included for many languages (not only English), like Danish, Dutch, English, Finnish, French, German and more.
• Wildcard search is supported (e.g. “xap*”).
• Synonyms are supported, both explicitly (e.g. “~cash”) and as an automatic form of query expansion.
• Xapian can suggest spelling corrections for user supplied queries. This is based on words which occur in the data being indexed, so works even for words which wouldn’t be found in a dictionary (e.g. “xapian” would be suggested as a correction for “xapain”).

Faceted Search

Xapian supports this added feature, called faceted search. What this basically means, is that search queries can have categories of interest and so can the terms and information indexed. This allows Xapian to narrow your search, to a particular sub-set of categories.

Performance:

• Supports database files > 2GB – essential for scaling to large document collections.
• Platform independent data formats – you can build a database on one machine and search it on another.
• Allows simultaneous update and searching. New documents become searchable right away.

Who uses it?

• Debian GNU/Linux
• Debian use Xapian for:
  • DebianMailingListArchiveSearch
  • Languages: Danish, Dutch, English, Finnish, French, German, Hungarian, Italian, Portuguese, Romanian, Russian, Spanish, Swedish, and Turkish
  • Database size: 3.9 million messages (2009-05-18)
  • DebianWikiSearch
  • Database size: over 6700 pages (2009-05-18)
  • Searchingtheirarchiveofsoftwarepackages
  • Database size: over 30000 packages (2009-03-16)

• LibraryoftheUniversityofCologne
  • Xapian is used withOpenBib to search the library’s OPAC (Online Public Access Catalogue)
• spin.de
  • Application: Free online community with profiles, chat, boards, blogs, games, and more

Sphinx

Our third item on the list is Sphinx, an open source full text search server. This one was designed for indexing database content. It currently supports MySQL, PostgreSQL, and ODBC-compliant databases as data sources natively.

The integration with RDBMSs (Relational Data Base Management Systems) allows us to use Sphinx as if it were a database itself. As it is with the rest of these solutions, this might be the thing that you need for your application or it can be completely irrelevant.

Some basic features:

Performance and scalability

• Indexing performance. Sphinx indexes up to 10-15 MB of text per second per single CPU core, that is 60+ MB/sec per server (on a dedicated indexing machine).
• Searching performance. According to their statistics, searching through 1,000,000-document (a 1.2GB text collection) runs at around 500 queries/sec on a 2-core desktop machine, with 2GB or RAM.
• Scalability. Biggest known Sphinx cluster indexes almost 5 billion documents, resulting in over 6 TB of data.

Key features

• Batch and Real-Time full-text indexes. Two index backends that support both offline index construction and incremental on-the-fly index updates are available.
• Non-text attributes support. An arbitrary number of attributes (product IDs, company names, prices, etc) can be stored in the index and used either just for retrieval (to avoid hitting the DB), or for efficient Sphinx-side search result set post-processing.
• SQL database indexing. Sphinx can directly access and index data stored in MySQL (all storage engines are supported), PostgreSQL, Oracle, Microsoft SQL Server, SQLite, Drizzle, and anything else that supports ODBC.
• Non-SQL storage indexing. Data can also be streamed to batch indexer in a simple XML format called XMLpipe, or inserted directly into an incremental RT index.
• Easy application integration.Sphinx comes with three different APIs:
  • SphinxAPI is a native library available for Java, PHP, Python, Perl, C, and other languages.
  • SphinxSE, a pluggable storage engine for MySQL, enables huge result sets to be shipped directly to MySQL server for post-processing.
  • SphinxQL lets the application query Sphinx using standard MySQL client library and query syntax.
• Advanced full-text searching syntax. It’s querying engine supports arbitrarily complex queries combining boolean operators, phrase, proximity, strict order, and quorum matching, field and position limits, exact keyword form matching, substring searches, etc.
• Rich database-like querying features. Sphinx does not limit you to just keyword searching. On top of full-text search result set, you can compute arbitrary arithmetic expressions, add WHERE conditions, do ORDER BY, GROUP BY, use MIN/MAX/AVG/SUM, aggregates etc. Essentially, full-blown SQL SELECT is supported.
• Relevance ranking. Sphinx claims to have a better-than-average ranking algorithm, analyzing keyword proximity, and ranking closer phrase matches higher, with perfect matches ranked on top. Also, ranking is flexible: you can choose from a number of built-in relevance functions, tweak their weights by using expressions, or develop new ones.
• Flexible text processing. Sphinx indexing features include full support for SBCS and UTF-8 encodings (meaning that effectively all world’s languages are supported); stopword removal and optional hit position removal (hitless indexing); morphology and synonym processing through word forms dictionaries and stemmers; exceptions and blended characters; and many more.
• Distributed searching. Searches can be distributed across multiple machines, enabling horizontal scale-out and high availability.

Some things to notice that are not provided by Sphinx:

• Wildcards: Sphinx (on it’s release version) does not allow the use of wildcards on it’s searches, although it’s latest beta version does.
• Sphinx cannot parse rich text format documents (like Word documents, PDF files, etc).

Who uses it?

• Craigslist: A classifieds page, with more than 20 billion page views per month.
• DailyMotion: Widely used video site.
• NetLog: Social network with more than 77million users.

Solr

Solr is an open source enterprise search platform from the Apache Lucene project. This means that it uses (and extends) the Lucene full-text search API we talked before, for all the full-text searches. This particular feature is of note, since this allows Solr to improve whenever the Lucene project integrates an improvement into its product. This is especially true as of May 2011, since both projects decided to merge.

It also differentiates from the other solutions presented here, because of it’s unique REST-like interface. You can put documents in it (called “indexing”) via XML, JSON or binary over HTTP and then you query them via HTTP GET and receive XML, JSON, or binary results.

This makes Solr completely language independent, allowing any developer using an HTTP capable programming language to interact with this platform.

Some basic features:

Searching:

• Advanced Full-Text Search Capabilities
  • External file-based configuration of stopword lists, synonym lists, and protected word lists
  • Many additional text analysis components including word splitting, regex and sounds-like filters
  • Sort by any number of fields, and by complex functions of numeric fields
  • Spelling suggestions for user queries
  • “More Like This” suggestions for given document

Caching

• Configurable Query Result, Filter, and Document cache instances
• Pluggable Cache implementations, including a lock free, high concurrency implementation
• Cache warming in background
  • When a new searcher is opened, configurable searches are run against it in order to warm it up to avoid slow first hits. During warming, the current searcher handles live requests.
• Auto warming in background
  • The most recently accessed items in the caches of the current searcher are re-populated in the new searcher, enabling high cache hit rates across index/searcher changes.
• Fast/small filter implementation

Admin Interface

• Full logging control
• Text analysis debugger, showing result of every stage in an analyzer
• Web Query Interface w/ debugging output
  • parsed query output
  • Lucene explain() document score detailing
  • explain score for documents outside of the requested range to debug why a given document wasn’t ranked higher.

Amongst others…

Who uses It?

• http://www.whitehouse.gov/ – Uses Solr via Drupal for site search w/highlighting & faceting (More details:1,2)a
• AOLis using Solr to power its channels
  • YellowPages
  • Music
  • NFLSports
  • AOLRecipes
  • ReferenceCenter
  • etc.
• digg uses Solr for search
• NASAPlanetaryDataSystem (PDS) uses Solr to search for dataset, mission, instrument, target, and host information
• reddit uses Solr for search
• And many more (http://wiki.apache.org/solr/PublicServers)