The dataset contains the html source and text content (true content) for around ~4000 webpages. One metric to measure your algorithm against this dataset could be the edit distance. If you do use this dataset, it would be great if you could share the results of your algorithms for benchmarks to compare against. I’ll be updating this post with the accuracy of our algorithm soon enough.

Download the dataset here (gzipped)

For the last couple of days, I and @sudheer_624 have been busy working on this hack for a Yahoo! Hackday. Although still a prototype, the hack has turned out to be interesting so we thought of putting it out for others to play around with.

Dygest (pronounced as ‘digest’ – thanks to @bluesmoon) is aimed at changing the conventional way of displaying search context via a snippet to a more informative, machine generated document summary. There two kinds of relevance for evaluating search results:

• Vertical relevance: determined by the ranking algorithms.
• Horizontal relevance: the contextual information made available to the user about the result – Searchmonkey is a good initiative on this front.

The current way of displaying this context is via a snippet of text under every result. This snippet shows the neighborhood of the occurrence of the query terms. Usually this information is not rich enough for a searcher to make the right judgement about the result. This causes the searcher to switch back and forth between the documents and the search results if the the page is not relevant. This can be frustrating at times.

Dygest aims to solve this by either replacing or enhancing the current search snippet with a summary of the result page. At its core lies a summarization engine which figures out what the *real* content of the page is (distinguishing it from the other junk like surrounding text, navigational text, comments etc) and then performs text summarization on this content. The summary of the page is then displayed to the user via the appropriate interface. How cool is that?

The user no longer needs to click on irrelevant links. He/She can perceive the theme/important facts of the page from right within the results page. The other advantage of this is that it gives the user a good overview of the query topic – he no longer needs to spend time reading many long documents but rather read a few summaries from the top results to get a good overview of the subject. This is particularly well suited for mobile devices where its frustrating to switch back and forth between pages and the search results. This is also fit for news articles where we just need the important facts about the story.

Well, here is an example to convince you. A search for ‘Carol Bartz’ yields the following result which at the first glance is not at all informative.

Enhancing the existing view with an abstract of the page helps gauge the content and theme of the document. This would now look like:

Dygest outputs the following summaries for the query ‘Iran‘ restricted to Yahoo! News:

And following for ‘Obama stimulus plan‘:

Currently, Dygest has two interfaces – (1) a search interface powered by yahoo boss and (2) a searchmonkey plugin. Its just a prototype so be kind and don’t be too judgmental.

Start dygesting here.

The following visualization depicts the dominant keywords/topics for one day (the terms are stemmed):

I had been reading a couple of Yahoo! related articles and visualization blogs. This is captured by the above visualization – but there is still alot of noise which I need to get rid of.

The next visualization depicts the linkages and clusters for the keywords. There exists a link between two terms if they occur in the same document. [may take sometime to load - you'll need to zoom in to get a better look - click on 'compute layout' if the clusters don't show]

Both the above visualizations depict important metrics that could be used to extract dominant themes from the browsing history. Dominance should not be just inferred from frequency but also from the prevalent of a term across multiple pages. I still need to work on removing noise and running this on larger datasets like browsing history for a week or so. If you have any ideas or good papers to recommend that would be nice.

President Eisenhower established the Advanced Research Projects Agency (ARPA) 50 years ago in response to the Soviet Union’s Sputnik launch, which surprised and embarrassed the United States as the Soviets became the first nation to successfully launch a satellite into space. DARPA’s 1958 charter charged the Agency to perform certain advanced research and development projects, with the primary mission of ensuring that the United States would never again be surprised by another nation’s technological advancement. The list of DARPA’s contributions includes the Saturn V rocket, the ARPANET (which laid foundations for the Internet).

On this very day 50 years ago, Feburary 7th 1958, the following short and concise document started it all: the original DOD directive.

Inspired by it I have decided to start logging my life as well. As of now its restricted to only my online life as I do not have resources like the SenseCam. The data collected in this process could be used in numerous ways: total recall, recommendations, predictions, and so on. As Peter Norvig says, “Its about the data and not the algorithm”.

Head over to the Life Logger project page, where I am documenting how I have been logging my life along with tools and algorithms for aggregating and analyzing such data.

Happy logging :-)

Click here for the Life Logger Project homepage

Whatsbuzzing is a destination site for online shopping. The site offers a one-stop service where consumers can browse across hundreds of storefronts, view the latest trends and find the hottest deals. In contrast to comparison shopping or product information sites, Whatsbuzzing provides visitors with the experience of a shopping mall. Visitors can browse storefronts by content, category or store name. A visitor can also tag storefronts so other consumers can find storefronts that are interesting. The storefronts are fully interactive and are constantly being updated with fresh content and timely offers.

As stated above, what makes it different from the plethora of shopping services is the unique content. Instead of just showcasing product details and prices, it also helps you keep track of the latest deals/discounts/offers – capturing the buzz in its true essence.

Another factor that makes it stand apart is its foray into being a browsing engine as compared to the omnipotent search engines. Although search is an integral part of Whatsbuzzing, it is just another feature to help assist the users to find products quickly.

It is surely the panacea to all my shopping woes. With the season of Christmas setting in why don’t you give it a try and come back with some feedback.

Sheeple is a term of disparagement, a portmanteau created by combining the words “sheep” and “people”; a reference to herd mentality. It is often used to denote persons who acquiesce to authority, and thus undermine their own human individuality.

Have we reduced to mere sheeples? Bloglines estimates a mere thousand feeds having greater than one subscriber. Quoting Eric Schmidt:

“The average blog has one reader: the blogger” [via]

When I glance through my Bloglines subscriptions I see the blogs of the biggies like Mike Arrington, John Battelle etc. Most of us view the blogosphere through this window (rather pin-hole), comprising of a few thousand popular blogs. Although their popularity speaks for their content, they restrict our perception of the blogosphere. There are numerous other bloggers who have a lot more valuable to say. But where is the audience?

There is no such service/app which gives us an unbiased view of whats happening and being said in the blogosphere. Its like the blogosphere has been virtually hijacked.

To get noticed, an anonymous blogger has to resort to either of the following:

• Become a (Sheeple) frequent reader of these blogs and participate by commenting hoping that someone will clickthru and notice you.
• Get featured in one of their posts (that is only if you are well connected).
• Write something that might offend them (probably this post won’t go unnoticed :) ).

So sheeples, transform yourselves back into people. Stop looking through the pin-hole and broaden your vision. You probably have no clue of how interesting the actual blogosphere is.

Paul Jardetzky and Quentin Stafford-Fraser setup up a video frame-grabber and hacked up a client-server program that would display an icon-sized image of the coffee pot, which refreshed at the rate of about three times every minute. This helped them to decide when the time was right to get a cup of coffee.

This image above is that of the actual coffee pot. The coffee machine was finally switched off at 0954 UTC on Wednesday 22nd August 2001.

It sometimes makes me wonder how small ideas/needs revolutionize the world.