Bruce MacVarish Notes

Expertise Analysis & Ranking with the SPEAR Algorithm

Michael G. Noll a CS Ph.D. candidate at Hasso Plattner Institute, Germany and Ching-man Au Yeung a PhD candidate in CC at the University of Southampton, UK have developed the SPEAR algorithm.

 

The SPEAR (Spamming-resistant Expertise Analysis and Ranking) is a new technique to measure the expertise of users by analyzing their public activities on platforms like Delicious.

From their post on the Delicious blog:

A major problem of the Internet today is that finding high quality information is not easy nor fast.  The steady increase of spam and junk content on the Web further complicates this challenge. Another related issue is that finding knowledgeable and trustworthy users on social platforms like Delicious is much more difficult than it should be. Wouldn’t it be nice if Delicious recommended “good” users with similar interests?

To tackle this problem, we created the SPEAR algorithm that analyzes the timeline of the bookmarking and tagging activities of users. The focus of SPEAR is on the ability of users to find new, high quality information on the Internet. A great benefit of SPEAR is that it returns two very useful sets of results: first, a list of users ranked by their expertise; and second, a list of websites ranked by their quality.

Technically, SPEAR is based on the well-known information retrieval algorithm HITS, a technique presented in 1999 that is used by search engines to rank Web pages. We came up with SPEAR by modifying HITS so that it fits to the characteristics of open and shared systems like Delicious and extended it with a new component that integrates the timeline of user activities into its analysis. This resulted in further performance improvements of the algorithm.

The two main elements of the new SPEAR algorithm are:

1. Mutual reinforcement of user expertise and document quality: A user’s expertise in a particular topic depends on the quality of the documents she or he has found, and the quality of documents in turn depends on the expertise of the users who have found them.

2. Discoverers vs. followers: Expert users should be discoverers – they tend to be faster than others to identify new and high quality documents. In other words, “the early bird catches the worm”. SPEAR gives more credit to users the earlier they find high quality documents.

The combination of both these elements has the effect that SPEAR favors quality over quantity of user actions, and that the algorithm is quite resistant to today’s spamming attacks.

We believe SPEAR is very useful in the context of open systems, particularly, social networks. That said, we are already researching the next version of the algorithm – the popularity of online services like Delicious is rising, and so is the spam threat. Whether we want to improve the user experience on Delicious or win the arms race against spammers, there’s still a lot of work left to do!

The Enterprise 2.0 "Edge Matrix" - 2009

In January of 2008, I created what I called the "Enterprise Edge Matrix" that mapped players who were targeting emerging opportunities at the edge of the enterprise.   

My hypothesis at the time was that enterprise core vendors, emerging start-ups and enterprise edge visionaries were racing to create new value for enterprise users at the Enterprise Edge.   To capture these new revenue growth opportunities at the "edge of the enterprise" , players need to develop new viral adoption strategies and networked business models that create and monetize value from data found in networks, markets and communities of enterprise end users.

At the time, I thought one interesting question was:

Who will be able to drive and monetize a viral enterprise application first?  Today's enterprise "core" leaders or the emerging enterprise "edge" visionaries?

Since early 2008, I've also outlined my vision of the Future State of the Enterprise Applications that can be summarized as:

- Applications delivered by Hybrid Cloud delivery models, 

- Business User driven with a focus on user experience 

- Social enabling transactions, conversations and relationship

- Device Agnostic applications running across devices, web and premise platforms

- Flow-ready by aggregating, filtering and visualizing user information and activity

- Sensing-based by acting on user's ambient and declarative data

So, based on the above, and reflecting industry evolution, investments, acquisitions and product announcements, I spent some time updating my Enterprise Edge Matrix for 2009.  

While I realize it needs some refinement, I believe the updated edge map highlights at least 3 significant trends from the past 12 months.

• Enterprise Core Vendors ( e.g. IBM, Cisco, Microsoft and Nokia) have aggressively moved to capture edge opportunities by introducing and/or acquiring new social, collaborative and mobile applications that directly target and are adopted by enterprise end users.  These solutions are branded and packaged as extensions to existing products and revenue streams.       

• Enterprise Edge Visionaries continue to add users but have not moved to monetize. 

• Emerging players are moving to leapfrog competitors with applications representative of Future State characteristics.  

Even with current economic conditions in mind,  applications that target and deliver directly to enterprise business users will see increased viral adoption and monetization.   

It would be interesting to hear how you'd refine this year's Edge Matrix... and what you think it will look like in early 2010.