SEO Articles
How to Make a WordPress Blog Duplicate Content Safe
November 30th, 2006
In one of my recent posts I wrote about the duplicate content issue. This topic is especially important to me since my blog uses the WordPress content management system which, when used with the default configuration, is not duplicate content proof. In fact this CMS is capable to render almost 100% of your content duplicate. As usual the fault of the system has roots in its advantages. WordPress has many features facilitating blogging and linking, such as RSS feeds to posts and comments, trackback URLs, monthly archives and so on. In the same time this variety of URLs returning similar or identical pages represents a clear case of duplicate content.
WordPress And Duplicate Content
The first evidences of duplicate content produced by your WordPress CMS can be found in your sidebar. They are category pages and monthly/daily archives. Category pages store your articles posted under the same topic – a category. Such pages have no unique content; they are just a collection of your previous posts. Monthly and daily archives also simply group your previous articles by the date of posting. Sometimes when you have only one post in a given day, the archive page for the date and your post are totally identical.
The next case of duplicate content is even more prominent. It can be your home page itself. If it contains not excerpts but the full text of your posts, then it duplicates your post pages. This also applies to the ‘next/previous entries’ pages – those accessible via /page/2, /3, /4 etc.
Feeds. Search engine spiders crawl all the content they can reach and of course this includes RSS feeds too. The additional problem with them is that Google may choose to display your RSS URL in the search results over the link to the original post. In this case the user who clicks this result will see an XML formatted page which is not ‘human-friendly’.
Trackback URLs. Many WordPress templates add trackback links after posts. This links enable authors to track who links to their posts. Usually, if your post URL looks like ‘www.yoursite.com/2006-11-30/yourpost/’ its trackback URL will be ‘www.yoursite.com/2006-11-30/yourpost/trackback/’.
Identical meta-description. By default WordPress doesn’t provide a tool to add unique meta description tags to your posts, and they either have none or share a single site-wide description. Having no meta description at all is a disadvantage, as a properly written one can make your snippet stand out in a SERP. Having an identical description for all your pages is a threat, as Google might get them filtered out as too similar. (see a thread here)
Because of the duplicate content Google search can return less desired URLs (such as feeds or archives instead of original posts); your pages can be moved out of their index, or placed into the supplemental results, which are rarely displayed to users.
Solving the Duplicate Content Issue in WordPress
Adding ‘noindex, follow’ tags
What can you do to avoid this problem? You can tell the search engines what URL to index by using ‘noindex, follow’ meta tag, robots.txt exclusions or 301 redirects. Let’s say you want Google to index your front page, posts, single pages and category pages and forbid the spiders from crawling the content of archives, feeds and ‘next entries’ pages - page/2, /3, … To do this you have to add to your header.php the following code:
if((is_home() && ($paged < 2 )) || is_single() || is_page() || is_category()){
echo '<meta name="robots" content="index,follow" />';
} else {
echo '<meta name="robots" content="noindex,follow" />';}
For those not familiar with editing templates in WordPress: in your dashboard click Presentation menu item and after the new page is opened – click Theme Editor. In the Theme Editor choose ‘header.php’ and then paste the above code into the editor form. This code has to be inserted anywhere between head tags .
Here the tag is added to the home page but not the ‘next entries’ page (is_home() and ($paged<2)), to your posts (is_single()); to solo pages, like ‘About me’, if you created any (is_page()); and to category pages (is_category()). If you don’t want your categories to be indexed just delete || is_category(). All the other pages will get . They will not be indexed, but this will not prevent crawlers from following their outgoing links.
Adding unique meta description
For this purpose I use Head Meta Description plugin. This plugin can be configured to use an excerpt of your post as a meta description – this is especially useful if you have to add this tag to hundreds of existing pages. Or you can add your own manually as a custom field, which is my personal preference.
Using more tag
By using this tag you tell WordPress to display only the first few lines of your post. This greatly reduces the similarity of home page and your articles. If you have too many existing posts to edit, you can use an ‘excerpt’ plugin, such as this one from Semiologic
Redirect to a canonical URL
You should edit your .htaccess file to perform 301 redirects. Non-www addresses like yoursite.com should be redirected to www.yoursite.com. URL without trailing slashes like www.yoursite.com/category should be rewritten to include it: www.yoursite.com/category/ This can be done by inserting the following code into your .htaccess file:
RewriteEngine On
RewriteCond %{HTTP_HOST} !^www\.yoursite\.com$ [NC]
RewriteRule ^(.*)$ http://www.yoursite.com/$1 [R,L]
RewriteBase /
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
For more details I advise you to read this: the process or rewriting the URL layout.
Preventing spiders from crawling feeds and auxiliary pages
For this purpose you should edit your robots.txt file by inserting the following code
User-agent: *
Disallow: /wp-
Disallow: /search
Disallow: /feed
Disallow: /comments/feed
Disallow: /feed/$
Disallow: /*/feed/$
Disallow: /*/feed/rss/$
Disallow: /*/trackback/$
Disallow: /*/*/feed/$
Disallow: /*/*/feed/rss/$
Disallow: /*/*/trackback/$
Disallow: /*/*/*/feed/$
Disallow: /*/*/*/feed/rss/$
Disallow: /*/*/*/trackback/$
Another two practical tips
Some people find it useful to restrict the number of posts displayed in your home page to 4-5, as less posts are duplicated.
A great article on customizing the more tag in Wordpress.
To Sum Up:
- To avoid the duplicate content issue in WordPress include you should do:
- Add ‘noindex, follow’ meta tag to your monthly/weekly/daily archives, ‘next entries’, and if necessary, category pages
- Ensure that all your pages have unique meta-description tags
- Set up 301 redirects for your non-www URL and URLs without trailing slashes
- Restrict search engine crawlers from indexing your feeds and trackbacks
- Use more tag to show excerpts in your home page instead of full posts
- Restrict the number of posts displayed in your home page
References:
- WordPress And Google: Avoiding Duplicate Content Issues thread in the WebmasterWorld forum
- Google indexing /feed URLs Issues thread in the WebmasterWorld forum
- 301 Redirect – a How-to by Beyondinc.com
- URL Rewriting Guide
Did you like it? Was it useful? Bookmark or share this post:
Subscribe for the Feed
Digg This!
Technorati
Del.icio.us
Furl
Blinklist
Ma.gnolia
Yahoo! My Web
76 Responses to “How to Make a WordPress Blog Duplicate Content Safe”
December 3rd, 2006 at 1:06 am
These are really good points to look upon and certainly I will implement few of them on my wordpress blog ASAP.
December 6th, 2006 at 8:56 am
Thanks, I found this usefull for my own blog, especially the extra robots.txt file additions and the canonical URL redirection script.
December 6th, 2006 at 9:15 am
Excellent article. I’ve been looking for a meta-description plugin. Thanks for point it out!
December 6th, 2006 at 3:28 pm
Thanx, well writen usefull resource!
B.t.w. assuming the code for the header index or not php will most likely be inserted it may make sense to include the php open and close tags as well as the closing } of the else block in the code example
Nonewithstanding just tagged this into stumble upon, hope you get lots of admiring traffic !
WordPress Rocks!
December 7th, 2006 at 8:11 am
Thanks Andre - indeed a lot of stumblers visited this page after you promoted it!
December 13th, 2006 at 5:37 pm
Kewl :-), I’m glad, its a good piece of work.
December 14th, 2006 at 12:20 pm
[...] » How to Make a WordPress Blog Duplicate Content Safe rest of the research blog also well worth a scan. Oh, and the author seems like a realy nice guy. [...]
December 15th, 2006 at 5:20 am
[...] In one of my recent posts I wrote about the duplicate content issue. This topic is especially important to me since my blog uses the WordPress content management system which, when used with the default configuration, is not duplicate content proof. In fact this CMS is capable to render almost […]Read full entry [...]
December 23rd, 2006 at 11:11 am
Cómo arreglar el problema del contenido duplicado en WordPress…
El tema del contenido duplicado se está comentando bastante por la blogosfera, — sin ir más lejos, yo mismo hace un par de días — y hoy voy a centrarme en WordPress.
Lo primero, decir que Online Marketing Research ha escrito How to Make…
December 28th, 2006 at 3:40 pm
[...] I am trying to clean up my duplicate WordPress content here at ecrosstexas.com. [...]
January 1st, 2007 at 2:25 am
thanks for the helpful post..i really appreciated…
January 4th, 2007 at 5:35 pm
Thank you for this! I’ve been looking all around for a decent article on the best methods relating to WordPress and spiders, and this is everything wrapped up in one:)
January 8th, 2007 at 5:29 pm
[...] Lo que mejor resume lo que he contado y recoge muy buenas aportaciones está en SeoResearcher . Nosotros empezaremos a probar todo esto desde ahora mismo y controlaremos los logs, el paso de robots por los blogs, indexación, etc y os iré contando si se solucionan los problemas. [...]
January 9th, 2007 at 3:24 am
[...] Os recomiendo los siguientes artículos relacionados a este tema: Sigt no dice Cómo arreglar el problema del contenido duplicado en WordPress. DupPrevent, Plugin para WordPress para controlar contenido duplicado. Fernando Gómez nos comenta sobre WordPress y las penalizaciones de Google por contenido duplicado. Seo researcher: How to Make a WordPress Blog Duplicate Content Safe. (enlace a cache de Google). Mariano nos habla también de El contenido duplicado en Google. [...]
January 30th, 2007 at 9:45 pm
[...] Here’s a run down of some strategies you can try. I’ve done the “noindex, follow” thing for my category pages on this blog. If you click on a category in the sidebar and do “view source” there is a meta tag instructing a search enging to not index this page, but follow the links so it finds the actual post pages and indexes them. [...]
February 20th, 2007 at 2:47 pm
I just found this article after getting nailed by G for duplicate content. I already knew about most of these things, but to have them all in one place made my job a lot easier. Thanks!
March 1st, 2007 at 3:36 am
Hi Andre,
This is great information. Thank you very much. I have been suffering from duplicate content in the SERPS and hoping that this will solve it!
Thanks for the valuable post.
March 1st, 2007 at 3:38 pm
Thank you very much fro your detailed input, this problem you are pointed out has been a big issue for our site
March 2nd, 2007 at 8:38 am
This a great article - very helpful stuff!
I’ve had a couple of first page searches completely dissapear off Google due to this duplicate content issue. Thanks for all the tips here.
A couple of questions:
1) This problem seems to be exclusively a Google problem (from my experience) - therefore in your Robot.txt suggestion, would it be wiser to specify Googlebot rather than blanket-disallowing all bots?
2) Do you have any information how long it takes for your SERPs to recover after you implement these changes? I want my traffic back
Thanks
March 14th, 2007 at 10:10 am
Thank you for this post, it’s great!
I agree with Andre SC, here’s an example from my Wordpress blog:
‘;
} else {
echo ”;
}
?>
I excluded Categories too. Insert this code between and in your header. Works great!
March 14th, 2007 at 10:13 am
Sorry, for some reason the code is invisible in the comment.
March 14th, 2007 at 5:02 pm
[...] Fear not though, there are ways to make WordPress search engine friendly. I’m not going to post a detailed technical ‘how to’ but I strongly recommend all WordPress users take a look at this excellent article on SEO Marketing Research. [...]
March 19th, 2007 at 9:12 pm
This is a great post - I’ll definitely be taking some of it into effect in the blogs we run.
March 21st, 2007 at 4:48 am
[...] Full version of the article including tutorial can be found here: Making Your Wo. (continues) [...]
March 22nd, 2007 at 8:24 am
[...] Langsung saya ambil dari sumbernya, maka tulisan berikut ini mungkin bisa membuat para pengguna wordpress semakin hati-hati dengan duplicate content atau situs anda akan miskin halaman di search engine seperti google yang nampaknya semakin ketat terhadap para plagiat. [...]
March 25th, 2007 at 6:09 am
good article - i have find a plugin to avoid dc
here the link: http://seologs.com/duplicate-content-cure/
March 29th, 2007 at 1:29 am
Question… I’ve implemented similar actions on my WP blogs before but in looking at your robots file I’ve thought of question in relation to side effects of not allowing gbot to index your feeds.
a ton of my traffic comes from google blog search. I should probably know this already, but where does G get their blog data from? are they watching the ping sites? are they watching feeds? obviously the post on the site is what is indexed, but what are they monitoring to get updates?
though I guess my sitemaps submission would get around any issues there. thoughts?
March 30th, 2007 at 11:58 am
great post - much of it is irritatingly obvious when you spend a minute thinking about it - but fantastic pulling it all together in one place.
Now just need to find the time to do this and the 101 other housekeeping tasks that blogging demands.
Best
cormski
April 6th, 2007 at 11:45 am
[...] Leyendo en un artículo de seorechearser.com titulado How to Make a WordPress Blog Duplicate Content Safe, encuentro más prevenciones para evitar el contenido duplicado en Wordpress. [...]
April 8th, 2007 at 4:19 pm
[...] Ausserdem ist mir aufgefallen, dass der Blog bei Google nur noch schlecht in den Suchergebnissen gelistet wird, vermutlich wegen “duplicate content”, weshalb ich hier einige Kleinigkeiten geändert habe, die dem einen oder der anderen Leserin aufallen werden. Mit was für einem Blödsinn man sich manchmal beschäftigen muss… var random_number = Math.random(); if (random_number < .5){ } else { } Möglicherweise verwandte Beiträge:Her mit den LinksLinktipps vom 10. 12. 20062. Geburtstag verpasstBiologische Uhr mit einfachem Test ablesbarSie entscheiden: Die besten Kinderwunsch-Blogs [...]
April 9th, 2007 at 5:22 am
Although Google says it would identify the duplicated content. I think it is always nice to make it easy for Google bot. As you mentioned, add ‘no-follow’ is a solution to dulplicate contents.
April 10th, 2007 at 4:27 pm
This is a great rundown of what needs to be done! Thanks for taking the time and gathering all the data you’ve done and/or what others are doing.
I’ve started to implement several of the tactics you’ve outlined above. I do wish there was an easier solution, like some sort of plugin for getting rid of the duplicants, based on the type of permalinks you select. Actually, I’m surprised no one has done that yet.
April 20th, 2007 at 5:59 pm
Hey,
I love what you’e doing!
Don’t ever change and best of luck.
Raymon W.
May 4th, 2007 at 7:35 pm
nice article friend…. I would use tips in this article to my blog site too.
May 11th, 2007 at 8:49 pm
Permalink can solve this duplication problem. Use the post id as a permalink then the search engines will see you less suspicious.
May 20th, 2007 at 2:02 pm
Great information. But, why not just use a specific \\\’no-follow\\\’ tag for Goggle?
San Diego RE
May 21st, 2007 at 3:24 am
Very interesting post, it clears up many issues about duplicated content in CMS and how to handle it in professional way. Again, thanks for this great post
May 23rd, 2007 at 1:15 pm
[...] Is WordPress is a problem? WordPress is giving you several capabilities of managing the content of your site and can be considered being some kind of content-management-system. But some of these features may cause trouble for you as WordPress could render almost 100% of your content as dublicate content. Oleg Ishenko of seoresearcher.com has written a very detailed article on how to make your wordpress blog duplicate content safe. It is definitely worth reading if you are running WordPress and are new to the duplicate content issue. [...]
May 25th, 2007 at 4:08 am
[...] There is a good post here on avoiding duplicate content issues with wordpress; many ideas are useful for other blogging platforms. [...]
May 27th, 2007 at 9:23 am
[...] This is the second part of the essential SEO tips for WordPress blogs covering the topics of Google Sitemaps plugins, pings and ping servers, valid (X)HTML, importance of a layout that puts post content ahead of sidebars and navigation, and displaying post excerpts and teaser text on the home page. You should also check out other articles relevant to the SEO for blogs: How to Make a WordPress Blog Duplicate Content Safe and SEO for WordPress Part 1 [...]
May 27th, 2007 at 11:21 pm
[...] *source-seoresearcher.com* [...]
June 19th, 2007 at 9:20 pm
Hello all
Very informative post. Please help this poor student with a BIG problem. I migrated from blogger to wordpress few days ago and I had 57 very successful,good,unique posts in Blogger and I moved those posts to Wordpress. Now if they reside on both the places - then its duplicate,right? How to avoid this? If I delete my blogger blog, I may take a lotta time to impress search engines!! Is there any way to ‘noindex’ only those 57 posts in my wordpress blog? Can I add a new category for those 57 posts and then noindex them in the header?
Thanks for reading this far!! Hope my issue is resolved
June 21st, 2007 at 12:12 pm
[...] Wer jetzt auf Nummer Sicher gehen will, sollte sich die Anleitung “How to make a WordPress duplicate content safe“. Ein erstes Grundgerüst liefert auch der robtos.txt Generator. [...]
June 29th, 2007 at 1:30 am
I’ve read there is a plugin that takes care of wordpress archives becoming duplicate content.
I do not recall it’s name however that would make things simpler.
July 1st, 2007 at 9:19 am
Very helpful, because Google looks for duplicate content and “bad” description and title-tags. Thanks.
July 7th, 2007 at 7:30 am
[...] Duplicate content can be major challenge for blogs and search engine traffic. Blogs by their nature create duplicate content on category and date archives, as well as on the blog home page. Search engines, especially Google, go to great lengths to keep duplicate content out of the search results, which means less traffic for you if they think your pages are repeating the same content. This issue is much too large to cover in depth here (for additional reading see the Google Webmaster Central Blog, SEOResearcher.com, and SEOmoz) but there are a few simple things that you can do to lower the risk of having your pages flagged as duplicates. [...]
July 26th, 2007 at 3:46 pm
Nice!
July 27th, 2007 at 4:28 am
[...] Salutes to the man - indeed a great article! [...]
August 22nd, 2007 at 3:17 pm
Just a short note to say I like your blog.
Good job and keep up the great work!
Kenneth
September 3rd, 2007 at 6:21 am
[...] Using recommendations like How to Make a WordPress Blog Duplicate Content Safe, my objective is adding the “noindex,follow” to every page except posts, home and categories pages. Paged home and categories pages (/page/2, …) should be tagged too. [...]
September 4th, 2007 at 9:08 am
[...] Is WordPress a problem? WordPress is giving you several capabilities of managing the content of your site and can be considered being some kind of content-management-system. But some of these features may cause trouble for you as WordPress could render almost 100% of your content as dublicate content. Oleg Ishenko of seoresearcher.com has written a very detailed article on how to make your wordpress blog duplicate content safe. It is definitely worth reading if you are running WordPress and are new to the duplicate content issue. [...]
October 22nd, 2007 at 9:52 pm
Great examples. How long will you keep the 301 Redirects in place? I would think that after 4 to 8 months there would no longer be a need for them.
October 23rd, 2007 at 12:24 pm
no follow tags, 301 redirects, sure seems like a bunch of work. I think there maybe a WordPress plug in that does all this for you.
November 13th, 2007 at 3:07 am
Im having a phobia adding noindex nofollow tag since blogger had the same problem with Google indexing previously.anyway its a good point im gonna try it. good tips man
November 20th, 2007 at 11:46 pm
[...] For tips how to get rid of the duplicate content in Wordpress please refer to my tutorial: Making Your Wordpress Blog Duplicate Content Safe [...]
December 1st, 2007 at 4:46 am
[...] I recommend you to read the post How to make your wordpress blog duplicate content safe, written by Oleg Ishenko, the owner of SeoSearcher.com. It’s an amazing read. [...]
December 10th, 2007 at 1:39 am
I’m using semiologic and wordpress as a cms, so I don’t have any issues with duplicate content on categories, archives, etc..
But what I found is that google is indexing search querys done in my site, so I have duplicate content there..
So I have urls indexed like http://www.domain.com/?s=query
Why is that happening, and how do I stop that?? Using robots.txt??
Great blog and good tips.. thanks
December 15th, 2007 at 6:44 am
very interesting, but I don’t agree with you
Idetrorce
December 20th, 2007 at 1:33 am
I would like to see a continuation of the topic
January 9th, 2008 at 3:11 pm
I’m getting ready to switch to Wordpress and this is interesting.
March 9th, 2008 at 8:18 am
Very informative post. It clears up many issues about duplicated content and how to handle it in professional way. Google looks for duplicate content, description and title tags.
March 19th, 2008 at 2:52 am
Very informative post. I will use some of the stuff for my blog. For your site it seems to work since it was the on top in my Google Search.
Thanks
March 21st, 2008 at 12:08 pm
Really very nice article on dup content. I really liked the way you have blocked the bot access from the .htaacess file.
April 3rd, 2008 at 1:45 pm
It’s good information. Still I would rather just write then fool around with all this code stuff. Really some one should figure a way to do this automatically.
thanks
May 3rd, 2008 at 10:03 am
[...] Rather than explain it all, Oleg Ishenko does a very good job showing How to Make a WordPress Blog Duplicate Content Safe with some very practical ideas to improve your situation with regards to search engine ranks. [...]
May 4th, 2008 at 5:38 pm
This article is great, I was having some indexing problems myself thanks to my feed getting indexed before my blog, this post is fantastic. Thanks for all the tips.
May 5th, 2008 at 10:26 pm
Just what I was looking for right now. These are excellent tips and very easy to understand. Thanks!
May 6th, 2008 at 10:33 am
That is very nice. Thank you very much.
May 14th, 2008 at 8:10 am
This article like me very well and was also very helpful. Thank you!
May 17th, 2008 at 4:22 pm
Great article. Thanks!
May 18th, 2008 at 2:02 pm
before starting a new project this post is like fresh air for me. thanks.
June 4th, 2008 at 5:19 am
I, too, had this article very helpful. Well it some good blogs.
June 10th, 2008 at 12:05 am
Thanks for the great info. I was really wondering how to deal with it and avoiding many unique pages to go to supplemental pages. Well, thanks for the info. I’d also tried category killer plugin, and I think it also worked, but couldn’t tell for certainity what is working as google is indexing my pages so slowly.
June 12th, 2008 at 12:59 am
[...] has very detailed and great post about this issue. Worth to check it out. This entry was posted on Wednesday, January 16th, [...]
July 22nd, 2008 at 9:41 pm
That was really a very interesting read. But I got a few questions to ask.
What about tag archives? Are they considered as duplicated content too? Should I prevent googlebots from indexing them as well?
August 7th, 2008 at 3:44 pm
Good points even if this post is a little bit old. I’ve been getting used to using Wordpress for a few months only but my question is similar to the commenter above.
What about tags? I see Google indexing not only my main pages but if I have 10 tags per post then that’s another 10 pages, theoretically, that Google is indexing but, it’s obviously duplicate content barring 1 word, the tag itself.
Any tips?
Thanks for the write up