The relative ease of creating and maintaining blogs makes them ideal tools for spamming search engines. Spam blogs or splogs serve two basic purposes: making money from advertising and affiliate programs, and participating in link farms. But making money from AdSense and providing nepotistic links are not what it takes to call a blog splog. Otherwise we would have to classify all blogs showing ads or promoting a business as spam; and there are thousands popular, quality blogs that would fall into this category. The distinctive feature of a splog, however, is that it has no use for its visitors. Should Google ban a splog from AdSense and prevent its links from passing on authority – such a splog would have no more value or purpose of existence. So my definition of a splog would be “a blog with the only purpose of showing contextual or affiliate ads, or boosting link popularity of certain target sites”.

How active are these splogs? This question calls for a little experiment; similar to one described by P. Kolari, A. Java and T. Finn in their paper “Characterizing the Splogosphere”. They did their experiment in early 2006, and I am going to repeat it at a smaller scale now, in the early 2007.

Every time a blog is updated it sends a ping to one of many ping servers in order to invite search engine crawlers to index the new post. I am going to use ping data provided by one of the most popular ping servers – Weblogs.com. Due to the limited scale of the experiment I will be using the smaller dataset covering the last 5 minutes of pings. It’s pretty big though: 8117 pings. I’ve written a simple Java application to parse the XML file and extract URLs and names of the blogs in the dataset. Also some of the blogs were classified by blog platform: Blogspot (Blogger), MySpace, Spaces.Live.com etc. I have discovered a number of popular blog services, that I haven’t come across yet, such as a popular Taiwanese site Wretch.cc, or Italian Libero.it and Splinder.com. I was surprised to see how few pings came from some other popular blog services; Livejournal for instance had only 6 pings! Obviously LJ doesn’t rely much on Weblogs.com, but LJ has little to do with my experiment, as it is known to have very small percentage of splogs.

So below is a break down of blogs by platform, according to a ping dataset retrieved on a Sunday evening, Feb. 11. Do not mix blogs under Wordpress.com category with blogs using WP as a blog engine. Only those blogs hosted by Wordpress.com are included into this category.

Fig. 1 Popular Blog Services in the Sunday Weblogs Dataset

The huge ‘Rest’ category consists of standalone blogs and blogs hosted by minor blog services.
A few words on the blogs in the dataset: a lot of blogs were not in English, I think as much as 70% of them. For instance, all Wretch.cc blogs and many Spaces.Live.com ones are in Chinese, there are also a lot of blogs in Italian, Spanish, Russian, Japanese and German.

Once dataset was downloaded and processed I started manually reviewing the blogs and discovering spam. Of course I couldn’t visit all the 8117 blogs, so I randomly selected 20 blogs from each category.

How did I classify spam blogs? While blogs with automatically generated content or dictionary dumps are easily classified as spam, those with plagiarized content or in foreign languages required a bit more of effort. Nepotistic links with keyword stuffed anchors were a good indicator of spam. Copyscape.com helped much discovering plagiarized posts. And finally, affiliate and contextual ads were the final complement in the spam classification problem. It has to be noted that very few blogs in languages other than English were classified as spam. I can be sure about my judgment of German and Russian blogs, since I know these languages, but when dealing with others I relied only on excessive advertising and nepotistic links as spam indicators. I skipped Wretch.cc and Explog.jp samples as I was totally unable to judge Chinese and Japanese blogs. In total of 177 reviewed blogs 36 were classified as spam.

Below you can see two charts, one indicating a ratio of spam within a sample, and another showing how much each blog platform contributes to the total amount of spam.

Fig 2. Percentage of Spam Blogs in 20-blogs Samples

Fig 3. Contribution of Each Category to the Total Blog Spam

With the notable exception of Blogspot, the majority of blogs hosted by popular blog services are spam free. Of course one can question their quality, as many of them are of little value to others. But let’s not forget that most of those blogs are private diaries or personal playgrounds never intended to have big audiences; and as long as they have value to the author and his/her close circle of friends we can’t call them spam.

Thus, according to my reviews blogs hosted by beon.ru, Libero.it, Spaces.Live.com, Livejournal.com, splinder.com, and typepad.com showed no instances of blog spam in 20 blogs samples. Among 20 MySpace blogs I have discovered 1 splog, and Wordpress.com sample contained 2. The popular Google’s service Blogspot has confirmed its unofficial name of Splogspot with 50% spam ratio. ‘The Rest’ category comprised by standalone blogs and blogs attached to commercial sites showed even bigger proportion of blog spam: 23 blogs of 27 reviewed were classified as spam. The relatively low number of splogs hosted by public services can be explained by anti-spam actions taken by the administration of such services. The standalone splogs, however, are not subject to such moderation, which allows them to thrive producing tons of junk content for SE crawlers and overloading ping servers with spam pings.

As you might have noticed I used the same style of charts introduced by the famous blog ModernLifeIsRubbish.co.uk, which has an excellent tutorial on how to create pretty pie charts in Adobe Illustrator. Highly recommended!

If anybody is interested, here is the dataset I used: Dataset

PhraseRank

From the very beginning, Google’s distinctive feature was the hyperlink induced popularity ranking. Algorithms using text content to evaluate relevancy of web documents played much lesser role. The reasons to this disparity are purely pragmatical: authors of web documents have total control over their content and are at liberty to modify it to deceive ranking algorithms and get higher positions in search results. Hyperlinks however are much less influenced by webmasters and provide a more reliable measure of authority (link weight) and relevance (link anchor).

Now Google introduces a new way to evaluate relevancy of a web document based on its content which might prove itself to be immune to manipulation attempts such as adjusting the keyword density or the automated generation of keyword-rich web pages. Actually the new system can become a remedy against MFA (Made For AdSense) sites that display meaningless scrapped keyword-rich content with paid contextual advertisements.

The new indexing and ranking system is based on the use of phrases. From a user’s point of view search queries in most cases are phrases or ‘concepts’, rather than sets of keywords. Despite this, conventional indexing systems still rely on individual terms. Indexing of phrases is avoided because the identification of all possible combinations of words would require immense computational and memory resources. For example a lexicon of 200,000 unique words could have approx. 3.2×10²⁶ phrases – with no system capable to store such a great amount of data in memory or efficiently manipulate it.

This problem is solved in the new system, which identifies phrases that are sufficiently frequent and distinguished in the crawled documents. By detecting phrases and indicating that they are ‘valid’ the system can identify multiple word phrases. This eliminates the need to index all the possible combinations of words in phrases that vary in length.

Another important feature is the ability of phrases to predict the presence of other phrases in a webpage. For example a phrase ‘President of the United States’ indicates that the document most likely contains the phrase ‘White House’. For every phrase the system creates a corresponding list of related phrases ordered according to their significance. This enables the system to detect spam pages based on the excessive appearance of related phrases.

So how does the system work?

Indexing

The process of indexing includes identification of phrases and related phrases. The system analyses the sequences of words and marks them as ‘good’ or ‘bad’ phrases. ‘Good’ phrases are those that occur quite frequently across the indexed documents or have a distinguished appearance, e.g. are delimited by markup tags, punctuation or other markers. Another distinguishing feature is the ability of a ‘good’ phrase to predict a related phrase – such as in above example ‘President of the United States’ predicts ‘White House’. Some phrases, for example, idioms (‘out of the blue’, ‘sitting ducks’ etc) tend to appear with different and unrelated phrases, and are not able to predict anything. Therefore idioms and colloquisms don’t count as ‘good’ phrases.

At the end of the indexing process the system produces a list of valid phrases along with a co-occurrence matrix as a predictive measure. An estimated size of the list is 650,000 phrases.

List of good phrases, or posting list has the following structure:

Phrase i: list:(document d, [list: related phrase count][related phrase information])

For each phrase i there is a list of documents d containing i. For each document there is the number of occurrences of the phrases related to i, and a bit vector containing the information about related phrases.

Bit vector consists of pair of bits. In each pair the value 1 in the first position indicates that a related phrase k is present in the document d; otherwise the value is 0. The second position indicates if a phrase l related to phrase k is present. The related phrases l of related phrases k are called ‘secondary related phrases of i‘. Bit vector is very important as it is used to determine relevancy of a document when the search results are ranked.

Example of a bit vector

Phrase i: document d: [related phrase counts:{3,4,3,0,0,2,1,1,0}]

related phrase bit vector:={11 11 10 00 00 10 10 10 01}

For phrase i there are 9 related phrases k. Now take a look at the bit vector. First pair indicates that both related phrase k₁ and one of its related phrases l are present in the document. Fourth and fifth pairs show that neither k₄ and k₅ nor their related phrases l are found, The last pair shows that although there is no occurrence of phrase k₉ one of its related phrases l is present.

For each phrase i the documents d are sorted in declining order according to the information retrieval-type score assigned to them with respect to the given phrase. This pre-ranking significantly improves performance of the system. To calculate ranking score the system can employ a link-popularity algorithm such as PageRank.

Phrase Identification. For a detailed description of the process please refer to [1] (paragraphs 0026 – 0102)

Searching

The search system receives a query and identifies phrases in it. Once the set Q of query phrases in created; the system retrieves posting lists for the query phrases in Q. Posting lists are intersected to determine, which documents appear on more than one list.

Phrase Based Document Ranking

Documents can be ranked according to their bit vector values. A document containing the most relevant phrases has the highest bit vector value and gets the highest ranking. Note that this approach uses the information about related phrases to rank search results, so even documents with low frequency of the query phrase q can get high rankings provided they have sufficiently high frequency of related phrases.

To produce the final ranking score the ‘body hit’ scores calculated above are combined with ‘anchor hit’ scores in a form of a linear function with adjustable weights, e.g.

Rank = (body hit score)*weight1 + (anchor hit score)*weight2.

For each phrase the indexing system also creates lists of documents in which the given phrase is an anchor in incoming and outgoing links. So the anchor hit score for document d can be calculated as a function of the related phrase bit vectors of the query phrases Q, where Q is an anchor term in a document that references document d.

Detecting Spam Documents

The new phrase based approach enables the future indexing system to detect and penalize spam documents. A statistical analysis of the document collection shows that normally a web page contains 8 to 20 related phrases. A spam document that deceives a search ranking system with an inflated keyword density is expected to contain an excessive number of related phrases, like 100 and more. Therefore by identifying deviations from the expected number of related phrases can be used to detect and battle spam in search results.

This system can also be applied to identify automatically generated content intended to be displayed along with paid contextual advertisements. Such sort of content is often used in MFA (Made for AdSense) sites and is nothing more than a meaningless sequence of keyword-rich text blocks scrapped from other websites, RSS feeds or search engine results pages. Although the conventional indexing systems are already quite effective in preventing these sites from showing in search results for popular terms, they still can occasionally appear in results for long-tail terms.

To Sum Up

The new indexing and ranking system proposed by Google uses page content (phrases) to rank search results in a way that is highly immune to manipulation attempts. The properties of a web document used to rank documents, i.e. phrases and relations between them, are influenced by the properties of all the other documents in the index, and therefore are out of control of webmasters.

The phrase based approach also enhances the ability of search engines to detect unnatural patterns in text content, such as inflated keyword density or scrapped content. It also enables search engine to provide more topically focused results by culling documents covering multiple topics.

The new approach can be used as an augmentation to the existing link-popularity based ranking systems as an additional parameter in the final score formula. Link popularity values are also used to pre-rank documents in posting lists to improve the performance of the search system.

Reference:

1. Patterson, A.L. “Detecting spam documents in a phrase based information retrieval system“, United States Patent Application, 12.28.2006

Penalties for having duplicate content can be really harmful. This is not just a downgrade in rankings but a move to supplementary results which are hardly visible to the most of the web users. Normally it is expected that Google would select one URL over another to display in SERPs, while duplicates could be found in supplemental results. Unfortunately this is not always so. In this thread [1] of the WebmasterWorld forum you can read about a case when an original high quality and authoritative page was removed from Google’s index together with its duplicates. Considering that this can happen even to the most honest webmaster, one can imagine the amount of attention this issue gets on any SEO forum.

Types of Duplicate Content

Duplicate content has a wider definition than the ‘copy-paste’ plagiarism; it is not just content scrapped from a competitor’s site, a SERP or a RSS feed. Apart from this there are few more aspects that are generally referred to as duplicate content.

Circular Navigation

Jake Baille from TrueLocal vaguely defines circular navigation as having multiple paths across website [2]. This can be understood as the same content being accessible via different URLs. An example of the circular navigation could be an article that is retrieved by links like
- www.example.com/articles/1/ ,
- www.mysite.com/article1/
- www.mysite.com/articles.php?id=1

Another legitimate use of multiple URLs is forum threads. Each thread can be accessible by a link like www.myforum.com/index.php/topic.1201.html , and each message within the tread has a URL like www.myforum.com/index.php/topic.1201.msg.01.html . In the eyes of a search engine all the links lead to different pages with identical content. Solution? Think of a consistent way of linking, or apply robots.txt exclusion rules.

This can also be the case when other people link to you using differently looking URLs. Since these external links are out of your control, you should create a 301 redirect to the canonical URL you choose to be displayed. A tutorial on 301 redirects can be found here [3].

.

Printer-Friendly Versions

Making a printer friendly version is a common practice and it adds value to the visitors. But printer-friendly version is also a prominent example of duplicate content! Fortunately a simple solution like adding a ‘noindex’ meta tag to your print pages solves the issue.

Product-Only Pages

Product pages looking similar are common among online stores. Typically they are created using a single template. Often two different product pages share a description that varies in just few words or numbers, which causes them to be filtered out as duplicate content. This issue has no easy solution. Either you rewrite robot.txt to allow only one product description to be crawled and lose SE traffic to the rest of them, or you roll up your sleeves and add something different to each product page, like testimonials, which is time consuming or nearly impossible depending on the number of product types in your stock.

How Do Duplicate Content Filters Work?

There are several algorithms in data mining aiming to detect similar text passages. The one claimed to be used by search engines [2] is w-shingling [4]. Each document has a unique fingerprint or shinglings – the contiguous subsequences of tokens (blocks of text). The ratio of magnitude of union and intersection of two documents’ shinglings can be used to determine their resemblance. Other algorithms that can be used for duplicates detection are Levenshtein’s distance [5] and Soundex [6].

It is naturally to expect from a duplicate content filter to be able to discover the origin and rank it higher. The simplest way to detect the origin would be comparing the date of indexing implying that the original source is uploaded and crawled earlier than its copies. But with the advent of the RSS feeds the new content can be distributed instantaneously and this approach is no longer valid.

Concerning the origin’s right to be ranked higher – this is not always implemented. In this article [9] you can read about an experiment of an article distribution. An article was syndicated twice scoring as many as 19000 copies. After some time Google, Yahoo and MSN have purged their indices leaving just few of the duplicates. MSN’s filter managed not only to discover the origin but also put it to the top of the search results. Yahoo has also discovered the origin, but in the results page to the title of the article, the origin’s position fluctuated obviously responding to the way Yahoo counts relevancy and authority.

To the tester’s amusement Google’s refined index did not include the original at all! Evidently Google featured only those pages with copies of the same article which it considered relevant and authoritative with no regard to the original source of the content! I’ve already mentioned a thread [1] where a similar problem is discussed. The both stories took place in 2005 and early 2006 and so far I found no evidence that this issue is resolved.

References and links to read about Duplicate Content

1. ‘Duplicate Content Observation‘. WebmasterWorld.com
2. ‘Duplicate Content Issues‘. SERoundtable.com.2006.02.28
3. ‘301 Redirect — a How-To‘ BeyondInk.com
4. ‘W-Shingling‘. Wikipedia
5. ‘Levenshtein Distance‘. Wikipedia
6. ‘Soundex‘. Wikipedia
7. ‘Duplicate Content Filter: What it is and how it works‘. WebConfs.com
8. CopyScape.com — discovers copied and similar pages.
9. ‘Duplicate Content Penalties Problems with Googles Filter‘ by J.S.Cassidy, published at SEOChat.com

“Black-hat” SEO is responsible for the immense amount of search engine spam — pages and links created solely to mislead search engines and boost rankings for client web sites. To weed out the web spam search engines can use statistical methods that allow computing distributions for a variety of page properties. The outlier values in these distributions can be associated with web spam. The ability to identify web spam is extremely valuable to search engine not just because it allows excluding spam pages from their indices but also using them to train more sophisticated machine learning algorithms capable to battle web spam with higher precision.

Using Statistics to Detect Search Engine Spam

An example of an application of statistical methods to detect web spam is presented in the paper “Spam, Damn Spam and Statistics” by Dennis Fetterly, Mark Manasse and Marc Najork from Microsoft. They used two sets of pages downloaded from the Internet. The first set was crawled repeatedly from November 2002 to February 2003 and consisted from 150 million URLs. For each page the researches recorded HTTP status, time of download, document length, number of non-markup words, and a vector indicating the changes in page content between downloads. A sample of this set (751 pages) was inspected manually and 61 spam pages were discovered, or 8.1% of the set with a confidence interval of 1.95% at 95% confidence.

Another set was crawled between July and September 2002 and comprises 429 million pages and 38 million HTTP redirects. For this set the following properties were recorded: URL, URLs of outgoing links; for the HTTP redirects – the source and the target URL. 535 pages were manually inspected and 37 of them were identified as spam (6.9%).

The research concentrates on studying the following properties of web pages:

• URL properties, including length and percentage of non-alphabetical characters (dashes, digits, dots etc.).
• Host name resolutions.
• Linkage properties.
• Content properties.
• Content evolution properties.
• Clustering properties.

URL Properties

Search engine optimizers often use numerous automatically generated pages to massively distribute their low PageRank to a single target page. Since the pages are machine generated we can expect their URLs to look differently from those created by humans. The assumptions are that these URLs are longer and include more non-alphabetical characters such as dashes, slashes or digits. When searching for spam pages we should consider the host component only, not the entire URL down to the page name.

Figure 1. Distribution of lengths of symbolic host names. Source [1]
See above the distribution of the length of the URL’s host component in the Set 2. The outliers in the right part of the chart are expected to contain spam pages. However the manual inspection of the 100 longest hostnames had revealed that 80 of them belong to adult site and 11 refer to the financial and credit related sites. Therefore in order to produce a spam identification rule the length property has to be combined with the percentage of non-alphabetical characters. In the given set 0.173% of URLs are at least 45 characters long and contain at least 6 dots, 5 dashes or 10 digits — and the vast majority of these pages appear to be spam. By changing the threshold values we can change the number of pages flagged as spam and the number of false positives.

Host Name Resolutions

One can notice that Google, given a query q, tends to rank a page higher if the host component of the page’s URL contains keywords from q. To utilize this search engine optimizers stuff pages with URLs containing popular keywords and keyphrases and set up DNS servers to resolve these URLs to a single IP. Generally SEOs generate a large number of host names to rank for a wide variety of popular queries.

Figure 2. Distribution of number of different host-names mapping to the same IP address. Source [1].

This behavior can also be relatively easy detected by observing the number of host name resolutions to a single IP. Take a look at the Figure 2 showing this distribution for the Set 2. For a better display the logarithmic scale is used. You can see that the majority of IP addresses are referred by 10 or less host names — the points in the upper left corner of the chart. So 1,864,807 IP addresses are mapped to only one host name, and 599,632 IPs — to 2 host names. In the lower right part of the figure you can observe some extreme cases with hundreds of thousands host names mapped to a single IP, and the record-breaking IP referred by 8,967,154 host names (the rightmost point).

To flag pages as spam a threshold of 10,000 name resolutions was chosen. About 3.46% of the pages in the Set 2 are served from IP addresses referred by 10,000 and more host names and the manual inspection of this sample proved that with very few exceptions they were spam. Lower threshold (1,000 name resolutions or 7.08% pages in the set) produces an unacceptable amount of false positives.

Linkage Properties

The Web consisting of interlinked pages has a structure of a graph. Therefore in graph terminology the number of outgoing links of a page can be referred to as the out-degree, while the in-degree equals to the number link pointing to a page. By analyzing out- and in-degrees values it is also possible to detect spam pages which would represent the outliers in the corresponding distributions.

Figure 3. Distribution of out-degrees. Source [1].

Figure 3 shows the distributions of out-degrees in the Set 2 with both axes drawn on a logarithmic scale. The graph is linear over a wide range, a shape characteristic of a Zipfian distribution. Notice the outliers in the blue oval. They show higher out-degrees than those expected according to the Zipfian distribution. For example there are 158,290 pages with out-degree 1301, while according to the overall trend only 1,700 such pages are expected. Overall 0.05% of pages in the Set 2 have out-degrees at least three times more than suggested by the Zipfian distribution, and according to the manual inspection of a cross section, almost all of them are spam.

Figure 4. Distribution of in-degrees. Source [1].

Similarly the distribution of in-degrees is drawn in Figure 4. Here there is even a larger portion of outliers. For example 369,457 pages have the in-degree of 1001, while according to the trend only 2,000 such pages are expected. Overall, 0.19% of pages in the Set 2 have in-degrees at least three times more common than the Zipfian distribution would suggest, and the majority of them are spam.

Content Properties

Despite the recent measures taken by search engines to diminish the effect of keyword stuffing, this technique is still used by some SEOs who generate pages filled with meaningless keywords to promote their AdSense pages. Quite often such pages are based on a single template and even have the same number of words which makes them especially easy to detect using statistical methods.

Figure 5. Variance of the word count of all pages served up bu a single host. Source [1].

For Set 1 the number of non-markup words in each page was recorded, so we can draw the variance of word count in pages downloaded from a given host name. The variance is plotted on the x-axis and the word count is shown on the y-axis, both axes are drawn on a logarithmic scale. Points in the left side of the graph marked with blue represent cases where at list 10 pages from a given host have the same word count. There are 944 such hosts (0.21% of the pages in Set 1). A random sample of 200 these pages was examined manually: 35% were spam, 3.5% contained no text and 41.5% were soft errors (a page with a message indicating that the resource is not currently available, despite the HTTP status code 200 “OK”).

Content Evolution

The natural evolution of the content in the Web is slow. In a period of a week 65% of all pages will not change at all, while only 0.8% will change completely. In contrast many spam SEO web pages generated in response to an HTTP request independent of the requested URL will change completely of every download. Therefore by looking into extreme cases of content mutation we search engines are able to detect web spam.

Figure 6. Average change week over weekof all pages served up by a given IP address. Source [1].

For this purpose the graph shown in Figure 6 was drawn for Set 1. The vertical axis shows the number of pairs of successful downloads from a single IP. Each pair of downloads was performed with 1 week interval to capture the content changes. The horizontal axis represents the grade of the content change: 85 — no change, 0 — total change. The outliers are marked blue and represent IPs serving the pages that change completely every week. Set 1 contains 367 such servers with 1,409,353 pages (97.2%). The manual examination of a sample of 106 pages showed that 103 (97.2%) were spam, 2 were soft errors and 1 adult pages counted as a false positive.

Clustering Properties

Automatically generated spam pages tend to look very similar. In fact, as already said above, most of them are based on the same model and have only minor differences (like inserting varying keywords into a template). Pages with such properties can be detected by applying clustering analysis to our samples.

To form clusters of similar pages the ’shingling’ algorithm described by Broder et al. [2] will be used. Figure 7 shows the distribution of the cluster sizes on near duplicate pages in Set 1. The horizontal axis shows the size of the cluster (the number of pages in the near-equivalence class), and the vertical axis shows how many such clusters Set 1 contains.

Figure 7. Distribution of sizes of clusters of near-duplicat documents. Source [1].

The outliers can be put into two groups. The group marked with red did not contain any spam pages, pages in this group are more related to the duplicated content issue. In the same time the group marked with blue is populated predominantly by spam documents. 15 of 20 largest clusters were spam containing 2,080,112 pages (1.38% of all pages in Set 1)

To Sum Up

The methods described above are the examples of a fairly simple statistical approach to spam detection. The real life algorithms are much more sophisticated and are based on machine learning technologies which allow search engine to detect and battle spam with a relatively high efficiency at an acceptable rate of false positives. Applying the spam detection techniques enables search engine to produce more relevant results and ensures a more fair competition based on the quality of web resources and not on technical tricks.

References:

1. Dennis Fetterly, Mark Manasse, Marc Najork. “Spam, Damn Spam, and Statistics: Using statistical analysis to locate spam web pages” (2004). Microsoft Research. Available at: http://research.microsoft.com/~najork/webdb2004.pdf

2. A. Broder, S. Glassman, M. Manasse, and G. Zweig. “Syntactic Clustering of the Web”. In 6th International World Wide Web Conference, April 1997.

Eye-Tracking Studies

The objective of eye tracking studies is gaining insight into how users browse the presented abstracts and select links to click. The results of eye tracking research provide Internet marketers with information on clickthrough rates, thus allowing them to make correct predictions on traffic changes as their rankings are gained or lost. For SE engineers the results provide a basis for improving the interfaces of search engines and metrics to evaluate the relevancy of the presented search results.

To detect users’ interaction patterns the eye tracking experiment observes a number of indicators of ocular behavior using a CCD (charged couple device) camera similar to the appliance used to read bar codes. The indices of ocular behavior include eye fixations, saccades, scan paths and pupil dilation. Eye fixations are defined as a stable gaze lasting for 200-300 milliseconds representing visual attention to a specific area of a SERP. Pupil dilations or pupil diameter changes represent a measurement of interest in a particular listing. This variable is especially important as it helps interpreting an implicit user feedback to the relevancy of the presented search results.

Cornell University Eye-Tracking Analysis of SE Users’ Behavior

One of the most recent eye tracking studies was performed at Cornell University by Laura A. Granka, Thorsten Joachims and Geri Cay ([1]). They used a sample of undergraduate students instructed to perform search in Google for 397 queries o topics covering movies, travel, music, politics, local and trivia. This study has produced the following results.

Fig 1. Google SEPR Click and Attention distribution ‘heat-map’

Study Results: Clicks and Attention Distribution

As you can see from the graph below and a SERP ‘heat-map’ based on it, the first two listings capture over a half of the user’s attention in terms of time of the eye fixation. Whereas the attention is shared almost equally, the difference in number of click between the first two listings is much more surprising: over four times! After the second listing the eye fixation drops sharply. Search results number 6 to 10 receive roughly equal attention. Here an interesting thing is that the 7th listing gets less attention than the succeeding 8th – apparently here we can observe the effect of the page fold. The 7th listing is just below the screen edge and is often skipped as users scroll the page down to the bottom (during the study the 7th listing was clicked only once). On the graph you can also see the 11th listing from the second page of the search results. It gets only about 1 percent of clicks and user attention – 2.5 times less than the lowest ranked result on the page one.

Fig 2. Time spent on viewing each results compared to the number of clicks. Source [1]

Often people consider getting to the ‘top-ten’ of Google as a measurement of the SEO success. Evidently this is a rather rough approximation. The ‘top-ten’ itself is a very diverse group with the number of clicks increasing almost logarithmically as your rank grows. For instance, the first five positions get over 88% of the traffic, and the first three – 79%.

SERP Browsing Patterns

Another important result of this study is the discovery of the browsing pattern: the way people read a SEPR. To assess the performance of the search algorithm it is vital to know how users evaluate the presented abstracts before clicking one of them. For example, if a user clicks the third listing, did he look the abstracts above and below it? The following figure shows how many results above and below of the selected listing are scanned on average.

Fig.3 Number of results scanned above and below the selected abstract. Source [1]

The effect of the page fold is clearly demonstrated here as well. While the first 5 listings are clicked after browsing through 1 to 2.68 listings above and below, the 7th listing is clicked after the entire page is examined! The listings below the page fold (8-10) are clicked after the first five or four listings are scanned. You can also see that the number of listings scanned above the clicked result is much bigger than the number of listings below. This indicates that users browse the list from top to bottom.

To Sum Up

While the study deals only with the first page of the organic search results, it can be assumed that similar results can be produced for other pages and perhaps even for the list of the paid ads in the right sidebar.

In addition to the academic researches there is a number of companies producing eye-tracking studies for the commercial use. The most notable of them are Eyetools.com and Poynterextra (http://www.poynterextra.org/EYETRACK2004/index.htm)

References:

1. Laura A. Granka, Thorsten Joachims, Geri Gay. ‘Eye-tracking analysis of user behavior in WWW search’, SIGIR, 2004. Available at http://www.cs.cornell.edu/People/tj/publications/granka_etal_04a.pdf Retrieved on 26.10.06

1. PageRank values range from 0 to 10.

While some people believe that PageRank is an integer number or at least converge to an integer after intensive recursive calculations, actually it is a floating point number. Google rounds up the real value to the closest integer and puts it on the 0-10 scale which is displayed in your browser toolbar.

2. PageRank value displayed in the toolbar is the one used to rank the results.

As you might have noticed, the toolbar value is updated every few months with no regular intervals. In the present time Google continuously calculates and updates PageRank so that sometimes actual PageRank and its toolbar values can differ. The toolbar value should be considered not as a current rank but as a level your page has reached by the time of the latest toolbar update.

3. PageRank is the primary factor to rank the search results.

Not exactly. PageRank was the backbone of the Google success as a search engine because of its integrity, ability to use the unique democratic nature of the web and hyperlinks, and relatively high immunity to abuse. But as years passed the Google technology became far more sophisticated. Now Google uses a cloud of factors to rank its search results. Some of them are query specific (keyword saturation of the page copy and the backlinks’anchor text) and some of them are domain specific (domain age, keywords in domain name, and of course PageRank). Nobody outside the Google’s offices knows the actual weight of each factor and it is quite possible that PageRank is no longer the primary one.

4. Google toolbar shows an increase of PageRank for my pages. My traffic is going to skyrocket!

Wrong. There won’t be any sudden traffic increase after toolbar upgrades any more. As I said before, the continuous calculation and update of the Google’s internal PageRank means that the rankings also adjust gradually as your pages get or lose backlinks. So the toolbar upgrade itself will not cause any changes in search results.

5. Toolbar PageRank is of no use, it is just for entertainment.

This is allegedly a quote by one of the Google representatives. This is only partially true. The reason why Google doesn’t show the actual PageRank any more is that there have been repeated attempts by hackers to access an exploit these data. Since 2004 the toolbar values updates are no longer synchronized with the actual rankings changes, and therefore should not to be considered too seriously in terms of SEO. However toolbar ranks still remains the easiest and most obvious way to evaluate the quality of a page and millions of web users regularly judge websites according to what Google toolbar shows them.

See also the following resources:

Matt Cutts explains PageRank

Can Google Multitask?

Google says: Toolbar PageRank is for entertainment purposes only

The idea behind AT(k) Algorithm is using only k highest authority weights instead of calculating average weight from every authority pointed by a hub. The parameter k is called authority threshold. A variant of an AT algorithm is MAX algorithm, where k=1, i.e. a hub is as good as the best authority it links to.

In general AT(k) algorithm uses the same formula as HITS. The difference is that when calculating the weight of a hub we consider top k authorities only, i.e. Fk(i) is a subset of outgoing links F(i). If the number of outgoing links |F(i)| is less or equal k than the AT(k) algorithm works exactly the same as HITS.

To overcome the shortcoming of the HITS algorithm of a hub getting a high weight when it points to numerous low-quality authorities, the following refinement was suggested. While using the same formula to calculate authority weights, the hub score h is now averaged by a number of outgoing links |F(i)|:

So in order to achieve a high weight a hub should link good authorities. Unfortunately this approach has its own flaw. Consider two hubs pointing to an equal number of equally good authorities. The two hubs are identical until one puts one more link to a low quality authority. The average sum of the authorities it points to sinks, and it gets penalized in weight. This is quite illogical but can be fixed by using so-called Authority Threshold Algorithm.

This algorithm was first described by Jon Kleinberg in his work “Authoritative Sources in a Hyperlinked Environment” (1998). The idea behind the HITS (Hyperlink Induced Topic Distillation) algorithm is that the authorities and hubs mutually reinforce each other. Authority weight of a page is calculated as a sum of hub weights pointing to it, and weight of a hub – as a sum of weights of authorities pointed to by it. In other words a hub is as good as the authorities linked by it, and vice versa.

The notation of the algorithm is as follows. Let S be a set of pages for which hub and authority weights are being calculated, n – the number of pages in the set. Then H is a subset of S containing pages acting as hubs, and A is a subset of S containing authorities. Since each page can be an authority and a hub, A and H overlap. For every page i in its hub role F(i) is the number of outgoing links. For every page i in its authority role B(i) is the number of incoming links. The n-dimensional vector of authority weights is denoted as a, and vector of hub weight – as h. Then hub and authority weights are calculated by the following formula:

The process is iterative. First all the weights receive value of 1. Then hubs and authority weights are calculated and the vectors are normalized. This stage is repeated until vectors a and h converge.

The algorithm however has a number of flaws. For example the nature of mutual reinforcement creates the following situation. Consider a hub that points to many authorities (hub B on the picture below), and a number of hubs pointing to a single authority (authority A on the picture). If the number of authorities pointed to by B is larger then the number of hubs pointing to A, then the HITS algorithm will allocate all the weight to the authorities in the right part of the picture and the authority A will get a weight near zero.

The reason is that hub B will initially get a very high score and propagate it to the authorities it links to. In the same time hubs on the left side will get very low score, and consequently A will get low weight too, although obviously it deserves more.

Cited Resources

• Kleinberg, J. May 1997, ‘Authoritative sources in a hyperlinked environment’. Technical Report RJ 10076, IBM,. Available at http://citeseer.ist.psu.edu/article/kleinberg98authoritative.html

HITS algorithm calculates hubs and authorities in query-time but relies on a relatively small subset of the Web – the immediate neighborhood of a page, since otherwise computation time would be unacceptably long. Hilltop algorithm analyses a query and calculates score values by finding pages that seem to be experts in the query-specific topic. This algorithm restricts itself to popular queries, since it can’t produce score values when no experts for an uncommon search term are found.

Topic-Sensitive PageRank extends the original PageRank idea by adding a query-time topic-sensitive adjustment. Instead of a single vector of PageRank values, multiple topic-specific PageRank vectors are calculated. Creation of a PageRank vectors for every possible topic would require extensive resources, so in practice the algorithm uses only 16 topic-specific ranking vectors representing the top categories of the ODP project. Other sources of topics can be used for this purpose as well, but since ODP project is created and edited by a large number of independent volunteers, it is the least likely to be influenced by any one party. For each page in the Web a set of importance scores with respect to various topics is precomputed and stored offline. In query time the topic-specific score is combined with other scores (e.g. content analysis) to form the final ranking for a page.

Let c_j be one of the 16 top-level ODP categories. For each topic c_j it is necessary to calculate a biased PageRank vector. Let M be a modified adjacency matrix. Each element m_ji has value 1/N_j, if there is a link from j to i, and where N_j is the number of outgoing links on page j. Otherwise element value is 0. Then the original PageRank formula in matrix notation looks as following:

Parameter α here is the dumping factor that equals 1-d in the original PageRank formula. The resulting vector of PageRank values is denoted as PR(α, p).

With minor modifications the same formula is used to calculate the topic-sensitive ranking vectors. Let T_j be the set of pages under a topic c_j. Then instead of the uniform distribution damping factor p, a non-uniform vector v_j is used, where:

Resulting PageRank vector is denoted as PR(a, v_ij). Additionally using all the documents under each topic c_j a term vector D_j is constructed. Term vector elements D_jt are the numbers of occurrences of every term under topic c_j. In order to detect a topic, to which a search query term relates to, two scenarios are considered. In the first scenario a user highlights a keyword in a page and initiates a search. In this case the search topic is defined by the page content. For example if word ‘architecture’ is highlighted in a page about famous buildings, the pages on CPU architecture should not appear among search results. So if a term q is highlighted in some page u, its context q’ would be the words in u. In the second scenario a user enters a keyword into a search form in the conventional way. In this case the context of the query q is the search term itself: q’ = q. When a history of search terms is kept, it is also possible to use it as the context q’. In query time the proximity of query context q’ to one of the topics c_j is calculated:

Proximity value P(q’|c_j) is calculated using the term vectors D_j. Then a query-sensitive score values are computed for every page d from the index that contains the original search term q. For each document d we sum up the products of topic proximity values P(c_j|q’) and the page rank r_d. The rank r_d is an element of the page rank vector PR(a, v_ji) of a topic to which the document d belongs to:

The final search results are sorted by the values of s_d. Since the PageRank calculations are performed in advance, the algorithm is able to quickly perform topic adjustments in the query time.

Cited resources

• Haveliwala, T.H. ‘Topic-sensitive PageRank’. In Proceedings of the Eleventh International World Wide Web Conference, Honolulu, Hawaii, May 2002. Available at http://citeseer.ist.psu.edu/haveliwala02topicsensitive.html
• Haveliwala, T.H. ‘Topic-sensitive pagerank: A context-sensitive ranking algorithm for web search’. IEEE Trans. Knowl. Data Eng., 15(4):784–796, 2003. Available at http://citeseer.ist.psu.edu/article/haveliwala03topicsensitive.html