Home > Blog > Google > SEO : Handy Dandy Surefire Robot Repellent Formulas – Don’t Let ‘em Mess With Your REP

SEO : Handy Dandy Surefire Robot Repellent Formulas – Don’t Let ‘em Mess With Your REP

Written By Noah Mallin | June 4, 2008 | 3 Comments

Robot T-Shirt

Filthy robots! There you are at the bar, just trying to lock eyes with a potential customer and those robots keep sliding their mecha-tendrils all over your back pages. At least it can feel that way sometimes in the online world. So much attention is paid to getting the attention of those ‘bots and droids that getting the right kind of attention is sometimes overlooked.

There are parts of your site that you might want robots to keep away from for better search optimization and indexing. Typical examples of these naughty bits include:

  • Duplicate content from printer friendly pages and dynamically rendered pages.
  • Confirmation and thank you pages that have little useful content.
  • Pages designed for internal use only (this would often be a test page or even a page with sensitive information that’s hosted on your site but not viewable by the general public.)
  • Form pages or login pages.

Bender

Thankfully there are a lot of options out there that websites can use to keep those robot’s oily fingers off of their junk. Yesterday Microsoft (full disclosure – a Reprise Media client) Yahoo! and Google all teamed up like some kind of supergroup of search to provide joint documentation on REP – Robot Exclusion Protocol. Here’s what they came up with and what it’s designed to do:

1. Robots.txt Directives

DIRECTIVE

IMPACT

USE CASES

Disallow

Tells a crawler not to index your site — your site’s robots.txt file still needs to be crawled to find this directive, however disallowed pages will not be crawled

‘No Crawl’ page from a site. This directive in the default syntax prevents specific path(s) of a site from being crawled.

Allow

Tells a crawler the specific pages on your site you want indexed so you can use this in combination with Disallow

This is useful in particular in conjunction with Disallow clauses, where a large section of a site is disallowed except for a small section within it

$ Wildcard Support

Tells a crawler to match everything from the end of a URL — large number of directories without specifying specific pages

‘No Crawl’ files with specific patterns, for example, files with certain filetypes that always have a certain extension, say pdf

* Wildcard Support

Tells a crawler to match a sequence of characters

‘No Crawl’ URLs with certain patterns, for example, disallow URLs with session ids or other extraneous parameters

Sitemaps Location

Tells a crawler where it can find your Sitemaps

Point to other locations where feeds exist to help crawlers find URLs on a site

2. HTML META Directives

DIRECTIVE

IMPACT

USE CASES

NOINDEX META Tag

Tells a crawler not to index a given page

Don’t index the page. This allows pages that are crawled to be kept out of the index.

NOFOLLOW META Tag

Tells a crawler not to follow a link to other content on a given page

Prevent publicly writeable areas to be abused by spammers looking for link credit. By using NOFOLLOW you let the robot know that you are discounting all outgoing links from this page.

NOSNIPPET META Tag

Tells a crawler not to display snippets in the search results for a given page

Present no snippet for the page on Search Results

NOARCHIVE META Tag

Tells a search engine not to show a “cached” link for a given page

Do not make available to users a copy of the page from the Search Engine cache

NOODP META Tag

Tells a crawler not to use a title and snippet from the Open Directory Project for a given page

Do not use the ODP (Open Directory Project) title and snippet for this page

Keep in mind that a link on another site to a page that uses REP can undo all that carefully applied robot repellent and send you back to square one. This is a rare occurrence but it does happen.

This entry was posted on Wednesday, June 4th, 2008 at 4:41 pm and is filed under Google, Microsoft, SEO, Technology, Yahoo!. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

«
»

3 Responses to “SEO : Handy Dandy Surefire Robot Repellent Formulas – Don’t Let ‘em Mess With Your REP”

  1. Search Engine Optimization Journal says:
    June 5, 2008 at 9:44 am

    Thanks for all the 411 on this. It’s tough when you’re at the bar and those damn robots go on and mess with ya! ;)

  2. Search Engine Optimization Journal says:
    June 5, 2008 at 9:44 am

    Thanks for the 411 on this. It’s tough when you’re at the bar and those damn robots start messin’ with ya! ;)

  3. Today’s 10 Most Interesting SEO News and Blog Posts | Marc Baumann says:
    June 5, 2008 at 6:51 pm

    [...] – SearchViews: Handy Dandy Surefire Robot Repellent Formulas [...]

Leave a Reply