Indexing the right things

Just a little FYI if you're running a blog site. You might want to limit the web search engines so that they only index your blog archives rather than your blog's index page.

I noticed after setting up my blog that I was getting a lot of hits from people searching for specific things but they were coming to the home page instead of the archived page where the content sits permanently. You see, more often than not, that content has already fallen off the main page, so it's better for them to find the data in the archive.

Here's an example: say you're searching for "brad choate trillian" using Google. Well, without restrictions, Google would have probably shown a match for that at www.bradchoate.com/index.php-- but chances are that the post I had made about Trillian would probably be in the archives already.

The solution to the problem is tell the search engines to not index your dynamic blog page, but to still follow links on the page and index them. Here's how you do it-- just place this within the <HEAD> portion of your HTML:

<meta name="robots" content="noindex, follow" />

That little rule will keep those search engines from indexing the wrong thing. If you tried that search for "brad choate trillian", you'd find that the match for it is in my archive directory as it should.

For more information about this meta tag, visit robotstxt.org.

2 Comments

A Google search for bradchoate doesn't turn up the front page of your site. Telling Google not to index your front page breaks I'm Feeling Lucky and makes it harder to get to your front page.

Google should notice that you update your site several times a week, and crawl accordingly. You don't need to push worry about old content not being on your front page because Google will quickly reindex your front page.

About

This article was published on February 25, 2002 2:40 PM.

The article previously posted was Visual Studio.net ad seen on....

The next article is DOCTYPE Syntax.

Many more can be found on the home page or by looking through the archives.

Powered by Movable Type