WordPress And Duplicate Content

The first evidences of duplicate content produced by your WordPress CMS can be found in your sidebar. They are category pages and monthly/daily archives. Category pages store your articles posted under the same topic – a category. Such pages have no unique content; they are just a collection of your previous posts. Monthly and daily archives also simply group your previous articles by the date of posting. Sometimes when you have only one post in a given day, the archive page for the date and your post are totally identical.

The next case of duplicate content is even more prominent. It can be your home page itself. If it contains not excerpts but the full text of your posts, then it duplicates your post pages. This also applies to the ‘next/previous entries’ pages – those accessible via /page/2, /3, /4 etc.

Feeds. Search engine spiders crawl all the content they can reach and of course this includes RSS feeds too. The additional problem with them is that Google may choose to display your RSS URL in the search results over the link to the original post. In this case the user who clicks this result will see an XML formatted page which is not ‘human-friendly’.

Trackback URLs. Many WordPress templates add trackback links after posts. This links enable authors to track who links to their posts. Usually, if your post URL looks like ‘www.yoursite.com/2006-11-30/yourpost/’ its trackback URL will be ‘www.yoursite.com/2006-11-30/yourpost/trackback/’.

Identical meta-description. By default WordPress doesn’t provide a tool to add unique meta description tags to your posts, and they either have none or share a single site-wide description. Having no meta description at all is a disadvantage, as a properly written one can make your snippet stand out in a SERP. Having an identical description for all your pages is a threat, as Google might get them filtered out as too similar. (see a thread here)

Because of the duplicate content Google search can return less desired URLs (such as feeds or archives instead of original posts); your pages can be moved out of their index, or placed into the supplemental results, which are rarely displayed to users.

Solving the Duplicate Content Issue in WordPress

Adding ‘noindex, follow’ tags

What can you do to avoid this problem? You can tell the search engines what URL to index by using ‘noindex, follow’ meta tag, robots.txt exclusions or 301 redirects. Let’s say you want Google to index your front page, posts, single pages and category pages and forbid the spiders from crawling the content of archives, feeds and ‘next entries’ pages – page/2, /3, … To do this you have to add to your header.php the following code:

     if((is_home() && ($paged < 2 )) || is_single() || is_page() || is_category()){
echo '<meta name="robots" content="index,follow" />';
} else {
echo '<meta name="robots" content="noindex,follow" />';}

For those not familiar with editing templates in WordPress: in your dashboard click Presentation menu item and after the new page is opened – click Theme Editor. In the Theme Editor choose ‘header.php’ and then paste the above code into the editor form. This code has to be inserted anywhere between head tags .

Here the tag is added to the home page but not the ‘next entries’ page (is_home() and ($paged<2)), to your posts (is_single()); to solo pages, like ‘About me’, if you created any (is_page()); and to category pages (is_category()). If you don’t want your categories to be indexed just delete || is_category(). All the other pages will get . They will not be indexed, but this will not prevent crawlers from following their outgoing links.

Adding unique meta description

For this purpose I use Head Meta Description plugin. This plugin can be configured to use an excerpt of your post as a meta description – this is especially useful if you have to add this tag to hundreds of existing pages. Or you can add your own manually as a custom field, which is my personal preference.

Using more tag

By using this tag you tell WordPress to display only the first few lines of your post. This greatly reduces the similarity of home page and your articles. If you have too many existing posts to edit, you can use an ‘excerpt’ plugin, such as this one from Semiologic

Redirect to a canonical URL

You should edit your .htaccess file to perform 301 redirects. Non-www addresses like yoursite.com should be redirected to www.yoursite.com. URL without trailing slashes like www.yoursite.com/category should be rewritten to include it: www.yoursite.com/category/ This can be done by inserting the following code into your .htaccess file:

RewriteEngine On
RewriteCond %{HTTP_HOST} !^www\.yoursite\.com$ [NC]
RewriteRule ^(.*)$ http://www.yoursite.com/$1 [R,L]
RewriteBase /
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]

For more details I advise you to read this: the process or rewriting the URL layout.

Preventing spiders from crawling feeds and auxiliary pages

For this purpose you should edit your robots.txt file by inserting the following code

User-agent: *
Disallow: /wp-
Disallow: /search
Disallow: /feed
Disallow: /comments/feed
Disallow: /feed/$
Disallow: /*/feed/$
Disallow: /*/feed/rss/$
Disallow: /*/trackback/$
Disallow: /*/*/feed/$
Disallow: /*/*/feed/rss/$
Disallow: /*/*/trackback/$
Disallow: /*/*/*/feed/$
Disallow: /*/*/*/feed/rss/$
Disallow: /*/*/*/trackback/$

Another two practical tips

Some people find it useful to restrict the number of posts displayed in your home page to 4-5, as less posts are duplicated.

A great article on customizing the more tag in Wordpress.

To Sum Up:

• To avoid the duplicate content issue in WordPress include you should do:
• Add ‘noindex, follow’ meta tag to your monthly/weekly/daily archives, ‘next entries’, and if necessary, category pages
• Ensure that all your pages have unique meta-description tags
• Set up 301 redirects for your non-www URL and URLs without trailing slashes
• Restrict search engine crawlers from indexing your feeds and trackbacks
• Use more tag to show excerpts in your home page instead of full posts
• Restrict the number of posts displayed in your home page

References:

• WordPress And Google: Avoiding Duplicate Content Issues thread in the WebmasterWorld forum
• Google indexing /feed URLs Issues thread in the WebmasterWorld forum
• 301 Redirect – a How-to by Beyondinc.com
• URL Rewriting Guide

Do you use Web 2.0 tagging websites? Well, I do. They are so much handier than the traditional browser bookmarks! I can easily access my favorites’ list at home, at the university campus, at cyber cafes, at airports or train stations. I can always find something interesting by looking at what other people tag. By now millions of Internet users have discovered the advantages of the social web sites like Digg, Fark, Del.icio.us and others. These websites bring huge benefits to webmasters as well. When your visitors tag your content they affirm that it is actually interesting and useful. And since most of the social websites allow favorites lists sharing, or use tags as a scoring parameter to rank news entries, thousands of other members can easily discover and visit your site. So social and tagging websites are indeed a great way to promote a site. By adding links to Web 2.0 tagging resources to your posts you remind the visitors to bookmark your content. Besides, including the post url and title as parameters into the link code simplifies the task of tagging.

How can you do it in your Wordpress blog? Quite easily. You don’t need to have PHP or CSS skills; although you have to understand what your Stylesheet and Main Index Template are for. We are going to create a panel of links that looks something like this:

I can’t guarantee that this tutorial will work with any WordPress theme – there are just too many of them. I used one of the simplest available (Simpla) hoping that this will ensure the maximum compatibility.

Web 2.0. Tagging Links Panel

First of all, we need these Digg, Technorati, Del.icio.us etc. icons. Download, unrar and copy them to your images/bookmarks/ directory.

Then we have to update your Stylesheet file. Login to your WordPress blog, and open the Dashboard. Click the ‘Presentation’ menu item and you will see your current theme as well as other available WordPress themes. Click the ‘Theme Editor’ link. On the Theme Editor page click ‘Stylesheet’ in the right menu, scroll the CSS code down and place your cursor at the end of the code, then press ‘Enter’ to start a new line. We are about to add a few lines of code to the stylesheet. Be careful not to alter or delete the existing code – this can ruin the outlook of your blog! To avoid an accidental damage of your theme, FIRST copy and paste the current stylesheet code to the Notepad and save it to your hard drive! Then copy and paste at the end of the stylesheet the following code:

Download file

Now click the ‘Update file button’.

To explain shortly why are we doing this: our links will be formatted with the UL tag. Normally the UL tag produces a list like this:

• item 1
• item 2
• item 3

But we need to have our links in one line. It can be done by inserting whitespaces between them, but this is not the best way to do it. By using the CSS modified UL tag we display our links in one line and ensure the equal intervals between them at any screen resolution.

Next step: adding links code to your WordPress template. I used some WordPress defined PHP functions to get the post url and post slug (the title of your post with whitespaces replaced by ‘+’ characters to be used as a parameter in tagging links). Also we are going to display the tagging links only when the post itself is displayed, but not on your front page. Otherwise the tagging panel would appear multiple times on your index page – after each post, and this can seriously damage the usability of your design. So I am going to use a simple is_home() check the ensure that this is indeed a post page.

In your Theme Editor click ‘Main Index Template’ item in the right menu. To avoid an accidental damage of your theme , FIRST copy and paste the current template code to the Notepad and save it to your hard drive! Our tagging panel will be shown right above the comments block. So look through the code and find the following line:

< div class="entrymeta">

Place your cursor right before this line and paste the following code:

Download file

Click the ‘Update File’ button.

The links contain the post url that is automatically retrieved by the_permalink() function, and the post slug is retrieved by the sanitize_title($post->post_title) function. When a user clicks on one of the links both parameters will be automatically entered into the submission form on the tagging website.

Now refresh your website. Does it look as it is supposed to? If yes – congratulations! If not – you’d better revert to the previous condition by copying and pasting the old versions of your theme that you saved before adding the above code blocks.

Once again: I cannot guarantee that this tutorial will work with any theme – there are just too many of them. I used one of the simplest available, hoping that this will ensure the maximum compatibility. If you have any questions about your particular case, comment this post, and we will try to find a solution.

Have fun!