I think that search is undergoing a paradigm shift – its no longer about who presents the best ten blue links but now more about presenting the answers upfront. Dfacto (pronounced as ‘de facto‘, Latin for ‘by [the] fact‘) is aimed at addressing this issue. A large percentage (nearly 68%) of queries are informational queries – one where the searcher knows what she’d like to do or find but does not know how this can be achieved. Dfacto is aimed primarily at addressing this class of queries by presenting a set of facts associated with the query/topic to the searcher. It uses natural language algorithms to get facts that are most “semantically” related to the query. In lay terms, it literally tries to understand your query and the results. I’ll save the algorithmic details for another post. The few examples below show how it works:

Disclaimer: This is a work in progress, so you might notice a few ‘facts’ that are irrelevant to the query.

Lets say the searcher is (losing hair and) looking for causes of hair loss. Normally he/she would need to click through a bunch of links to get an overview on the causes. This hack on the other hand makes life a bit easier by presenting the causes upfront (click to enlarge):

click to enlarge

Along with the facts, we also list the source from where it was extracted. Alternatively, the searcher can also select a bunch of facts he/she thinks are relevant and refine the search. This in turn would yield a new set of ‘web results’ along with new refined and related ‘facts’.

Another example (one which I particularly like) is a query about ‘table manners’. This precisely lists a set of etiquette’s to follow at the table (click to enlarge).

click to enlarge

Alternatively, Dfacto also serves well as a product research tool. A query for ‘iphone 3gs’ yeilds (click to enlarge):

click to enlarge

On another note, if you have a date in the coming weeks you might be interested in reading the list below (:

click to enlarge

Happy hacking!

For the last couple of days, I and @sudheer_624 have been busy working on this hack for a Yahoo! Hackday. Although still a prototype, the hack has turned out to be interesting so we thought of putting it out for others to play around with.

Dygest (pronounced as ‘digest’ – thanks to @bluesmoon) is aimed at changing the conventional way of displaying search context via a snippet to a more informative, machine generated document summary. There two kinds of relevance for evaluating search results:

• Vertical relevance: determined by the ranking algorithms.
• Horizontal relevance: the contextual information made available to the user about the result – Searchmonkey is a good initiative on this front.

The current way of displaying this context is via a snippet of text under every result. This snippet shows the neighborhood of the occurrence of the query terms. Usually this information is not rich enough for a searcher to make the right judgement about the result. This causes the searcher to switch back and forth between the documents and the search results if the the page is not relevant. This can be frustrating at times.

Dygest aims to solve this by either replacing or enhancing the current search snippet with a summary of the result page. At its core lies a summarization engine which figures out what the *real* content of the page is (distinguishing it from the other junk like surrounding text, navigational text, comments etc) and then performs text summarization on this content. The summary of the page is then displayed to the user via the appropriate interface. How cool is that?

The user no longer needs to click on irrelevant links. He/She can perceive the theme/important facts of the page from right within the results page. The other advantage of this is that it gives the user a good overview of the query topic – he no longer needs to spend time reading many long documents but rather read a few summaries from the top results to get a good overview of the subject. This is particularly well suited for mobile devices where its frustrating to switch back and forth between pages and the search results. This is also fit for news articles where we just need the important facts about the story.

Well, here is an example to convince you. A search for ‘Carol Bartz’ yields the following result which at the first glance is not at all informative.

Enhancing the existing view with an abstract of the page helps gauge the content and theme of the document. This would now look like:

Dygest outputs the following summaries for the query ‘Iran‘ restricted to Yahoo! News:

And following for ‘Obama stimulus plan‘:

Currently, Dygest has two interfaces – (1) a search interface powered by yahoo boss and (2) a searchmonkey plugin. Its just a prototype so be kind and don’t be too judgmental.

Start dygesting here.

This is a follow-up post of The Monkey Just Got Delicious – I. The app is not yet public for the reasons mentioned in part I. As I had mentioned, my goal was to generate a tag cloud for the search results. Well, search monkey does not allow you to spit out arbitrary html, thus making it difficult to render a tag cloud. After much thought I settled for a color coded tag cloud (as in the screenshot below). You will notice the color of the tags gradually fading (darker shade means that the tag is more popular).

Got feedback, will listen.

[UPDATE] New tag cloud UI for deliciousify can be viewed here

[UPDATE] The search monkey app is currently disabled for public use as it was hitting the delicious rate limit. Hence it will remain as a prototype for now. BTW the delicious team is working on their own search monkey app and I bet its going to be much cooler.

I’m a big fan and an avid user of Yahoo! Search Monkey. So this weekend I decided to write myself a search monkey application that I have always wished for. Well, we all will agree that nothing beats human created metadata and what better metadata about search results can there be than the vast and rich knowledge stored in bookmarking services. My search monkey application deals with enriching the organic search results from Yahoo! with metadata from del.icio.us.

Sometimes the search summary does not provide a useful insight into the contents of the search result (as seen below). The only way users ascertain relevance is by clicking on the result and figuring it out themselves. Wouldn’t it be better if the contents of the result could be summarized by just a few words – keywords that highlight broadly what the document talks about. Deliciousify (as seen below) aims to solve this problem by listing the top tags about a search result from del.icio.us, along with its popularity (number of people who have bookmarked it). In the future, I plan to display a tag cloud for the results. Give it a try and send any comments/feedback my way.

I’m trying to consolidate all such performance centric blogs at Performance Juice. While I’m busy scouring the web for some great blogs I may have missed, I would love to include any links you might have into Performance Juice. You can contribute by listing blogs into the spreadsheet below. You can edit the spreadsheet here. Lets make Performance Juice a community driven vertical search for anything performance.

Be sure to quench your performance thirst at Performance Juice or just subscribe to this feed and stay up to date on the performance karma.

There are many performance related posts we come across everyday which are not necessarily from performance centric blogs. If you come across any such great posts, just tag them as “performancejuice” on del.icio.us and I’ll make sure they are included.

Any volunteers to manage the search engine? Head here.

[UPDATE] The custom search is now also powered by performance bookmarks on del.icio.us.

You can edit the spreadsheet here.

< META name='unavailable_after' content='Wed, 01 Aug 2007 00:00:01 GMT'>

By specifying this tag, webmasters can tell Google not to index a particular page after the specified date OR consider the given page as stale. This would be appropriate for promotional pages where the promotions expire after a given time. This would help unclog the search engine indexes of irrelevant data.

This valuable piece of information, provided by the ‘unavailable_after’ tag, would not only be used to clear up the Google’s index but could also make its way into Google’s ranking algorithms. There are two perspectives to how a search engine could use this data for ranking:

• When a page specifies its expiry/unavailability date, it implicitly tells the search engine of the period for which it would be most relevant. Hence, as the unavailability date of a page approaches it should start becoming less irrelevant to a users query. For example: The user query for ‘fedora release notes‘, currently, has the top 3 results pointing to the notes of FC7, whereas the other results have a random mix of release notes for FC3, FC2, FC4 and FC5 (with FC3 and FC2 pages being ranked higher than FC5 and FC4 respectively). Lets say that FC8 was going to be released this November [schedule]. Assume that the Release Notes page for FC7 has the unavailable_after tag set for sometime around December. Thus, as December approaches, FC7 pages would start losing their relevancy for such queries and gradually transition lower in the search rankings, making FC8 the most relevant result. This would resolve the current inaccurate ordering of results obtained for the query ‘fedora release notes‘ on Google. This could be achieved in a manner similar to proposed in the following paper: Time Damping Of Textual Relevance.
• This perspective (inverse to the above perspective) would be very specific to promotional/shopping related pages. Most shopping promotions/offers are valid for a given period. Hence, such pages should become more relevant to a users query as they approach their expiry date. For example: Consider a user query for ‘20% discount shoes‘. Lets assume that the results have pages from Zappos as well as Shoebuy both offering a 20% discount on shoes. The Shoebuy sale is going to last for about two more weeks from today (as specified by the unavailable_after tag) whereas the Zappos sale would be ending in another two days. Since both the stores are offering the same percentage discount, it would be more appropriate to rank the Zappos page higher as its offer would be ending soon. From the point of a user (shopper), he would be more interested in looking at offers ending soon, as he can always checkout the other long lasting offers at some later time.

The perspectives above represent only a few of the conceivable usages of the unavailable_after tag. There could be a numerous other perspectives to how this data could be utilized to improve search rankings.

I would lover to hear your take on the unavailable tag, particularly if you can provide another perspective to utilizing this data.

Conventional Methods

Most systems, that allow tag search, base their rankings on factors like simply the ‘number of unique users’ or on ratios like ‘number of unique users for tag t / number of unique users for all tags’ etc. These conventional algorithms do work, but not quite so well for large datasets where they can be exploited. They also often do not represent the true relevance. Reminds me often of the pre-PageRank era of information retrieval systems.

So, which relevance algorithm do I use?

Well, you can always use the conventional methods, but then you can always try the algorithm I devised. This algorithm seems to capture the true essence of relevance in tagging systems. I call it the WisdomRank as it is truly based on the ‘wisdom’ of the crowds, the fundamental part of any social system. Read along to understand it in detail (or download the pdf).

Inferring relevance for tag search

from user authority – Abstract

Tagging is an act of imparting human knowledge/wisdom to objects. Thus a tag, a one word interpretation/categorization of the object by the user, fundamentally represents the basic unit of human wisdom for any object. This wisdom is difficult to quantify as it is relative for every user. One approach to quantify this would be to use the wisdom of the other users to define this for us. This can be done by assuming that every tag corresponds to a topic for which every user has some authority. Also, every tag added to an object corresponds to a vote, similar to the Digg model, asserting that the object belongs to that topic (tag).

Let us consider a user Ui who has tagged object Oj with the tag Tk. Whenever other users in the system tag Oj with Tk, they are implicitly affirming Ui’s wisdom for tag Tk.

Thus, we define the function affirmation for the tuple(u, d, t) as the number of other users who have also tagged document d with tag t:

affirmation(u, d, t) = ∑_{i=All users except ‘u’} tagged(u_i, d, t)

where,

u – the user
d – the document/object
t – the tag
tagged – 1 if the user Ui has tagged d with t
- 0 otherwise

Hence, we can proceed to define the wisdom of the user for a topic (tag) t as the sum of all such assertions by other users,

wisdom(u, t) = ∑_{x=For all documents d tagged with tag t by U} affirmation(u, d, t)

Likewise, we can now define the authority of a user for the topic t, as the ratio of the user’s wisdom to the collective wisdom for t. Hence,

authority(u, t) = wisdom(u, t) / ∑ wisdom(u_i, t)

For example: Let us determine the authority of user u1 for tag t1

Object d1: Object d2: Object d3:
t1 by u1 t1 by u1 t1 by u2
t1 by u2 t3 by u1 t1 by u3
t1 by u3 t3 by u1
t2 by u1

affirmation(u1, d1, t1) = 2 affirmation(u1, d2, t1) = 0
Hence, wisdom(u1, t1) = 2

Likewise for other users,

wisdom(u2, t1) = 3
wisdom(u3, t1) = 3

Hence the authority of user u1 for t1 is as follows:

authority(u1, t1) = 2 / (2 + 3 + 3) = 2 / 8 = 0.25

Whenever a user tags an object with a tag, he does so with the authority he possesses for that tag. Thus as compared to conventional methods, where the objects are usually ranked on the number of instances of the tags, in this method the measure of the relevance of a tag for an object is equivalent to the sum of all such user authorities. Thus,

relevance_metric(d, t) = ∑_{i= all user who have tagged document d with t} authority(u, t)

This relevance score, when calculated for every tag would provide an accurate measure for ranking the objects. As compared to the conventional methods where more number of instances of a tag for an object ensured a higher relevance for that tag, here the number of authoritative users counts.

Let us consider the following example:

Object d1: Object d2:
t1 by u1 t1 by u2
t2 by u5 t1 by u3
t1 by u4

Let us assume that u1 has a very high authority for tag t1. Hence in the above scenario, a search for tag t1 may rank d1 higher than d2, if

authority(u1, t1) > authority(u2, t1) + authority(u3, t1) + authority(u4, t1)

This result is with the assumption that u1’s authority is greater than those of u2,u3 and u4 combined.

On the other hand, d2 would be ranked higher than d1 if the combined authorities of u2, u3 and u4 exceed that of u1. If the majority of the users are suggesting something, it indicates that their suggestion is far more valuable than that of an individual user or a subset of users.

Future Enhancements

While calculating the user assertions this algorithm currently considers all such users as equal even though they may have varying authorities for the corresponding tag. As a future enhancement, I plan to incorporate the authorities of the users as well into the affirmation calculations.

So, what is cosine similarity?

Those not acquainted with Term Vector Theory and Cosine similarity can read this article.

Why does cosine similarity fail to capture the whole picture?

Let us consider two documents A and B represented by the vectors in the above figure. The cosine treats both vectors as unit vectors by normalizing them, giving you a measure of the angle between the two vectors. It does provide an accurate measure of similarity but with no regard to magnitude. But magnitude is an important factor while considering similarity.

For example, the cosine between a document which has ‘machine’ occurring 3 times and ‘learning’ 4 times and another document which has ‘machine’ occurring 300 times and ‘learning’ 400 times will hint that the two documents are pointing in almost the same direction. If magnitude (euclidean distance) was taken into account, the results would be quite different.

How do I get the accurate measure of similarity?

We have at our disposal two factors: one the cosine which gives us a measure of how similar two documents are, and the second the (euclidean) distance which gives us the magnitude of difference between the two documents. There could be a number of ways you could combine the two to determine the similarity measure.

Conclusion

The magnitude and cosine both provide us with a different aspect of similarity between two entities. It is upto us to either use them individually or in unison (as above) depending upon our application needs.

[Update]

As pointed out be Dr. E. Garcia (in the comments), similarities can be expressed by cosines, dot products, Jaccard’s Coefficients and in many other ways.

]]> http://semanticvoid.com/blog/2007/02/23/similarity-measure-cosine-similarity-or-euclidean-distance-or-both/feed/ 32 http://semanticvoid.com/blog/2006/12/11/whats-the-buzz-in-the-shoposphere/ http://semanticvoid.com/blog/2006/12/11/whats-the-buzz-in-the-shoposphere/#comments Mon, 11 Dec 2006 15:16:11 +0000 Anand Kishore http://semanticvoid.com/blog/2006/12/11/whats-the-buzz-in-the-shoposphere/ That certainly was a tough question to answer, but not anymore. Whatsbuzzing, which released a few days back (reminds me of the sleepless nights :-)), helps you do just that. The brainchild of Anand Jagannathan, Whatsbuzzing is aimed at solving the online shopping woes of users. As described in Anand’s blog:

Whatsbuzzing is a destination site for online shopping. The site offers a one-stop service where consumers can browse across hundreds of storefronts, view the latest trends and find the hottest deals. In contrast to comparison shopping or product information sites, Whatsbuzzing provides visitors with the experience of a shopping mall. Visitors can browse storefronts by content, category or store name. A visitor can also tag storefronts so other consumers can find storefronts that are interesting. The storefronts are fully interactive and are constantly being updated with fresh content and timely offers.

As stated above, what makes it different from the plethora of shopping services is the unique content. Instead of just showcasing product details and prices, it also helps you keep track of the latest deals/discounts/offers – capturing the buzz in its true essence.

Another factor that makes it stand apart is its foray into being a browsing engine as compared to the omnipotent search engines. Although search is an integral part of Whatsbuzzing, it is just another feature to help assist the users to find products quickly.

It is surely the panacea to all my shopping woes. With the season of Christmas setting in why don’t you give it a try and come back with some feedback.

content:neural network AND (category:biology OR category:AI)

Everything seems to work fine. Well not yet. In the above query we are trying to retrieve documents contaning the words ‘neural network‘. But if you look closely (try getting an explanation of the score in lucene), although the category sub-query seems to be used only for limiting the range of documents to particular domains, it contributes to the relevance as well.

So you must be wondering “How do I get documents from Biology or AI with ranking based on their relevance with ‘neural network’?”. Here is how. You dont need to hack around the lucene source code. All you have to do is to give a nullifying boost (thats a cool oxymoron (-;) to the respective sub-query. By nullifying boost I mean, a boost value so small that in effect nullifies the score of the sub-query (something like 0.00001). Therefore the revised query would look like:

content:neural network AND (category:biology OR category:AI)^0.00001

Thus although the category sub-query is a must match for a document, inorder to be a part of the resultset, it does not contribute to the score of the document. I like to term such queries non-relevant booleans. Non-relevant as it does not contribute to relevance and boolean as in the condition (AND or OR) as per which it must match.

This lets us harness the querying capabilities of a database from within a search engine.

[UPDATE] A nullifying boost of zero would be the ideal case wherein you don’t want the subquery to contribute to the score at all. A non-zero value for the same would give you more control over the subquery’s contribution to the score.