Meta Robots Tag
Tag: <meta name=”robots”>
Syntax: <meta name=”robots” content=”xxx”>
The meta robots tag includes directives for the search engine spiders. “xxx” is a comma-separated list of following directives:
The robots Meta Tag is used to restrict search engines indexing of a Web page or following links on that page. In addition, you can instruct the various search engines to include your description in their SERP’s over the one they might automatically generate.
• This tag is an alternative to the more functional robots.txt file usage. The difference is that robots.txt file in addition to control over indexing pages and crawling links on a page helps you to block the search engine spiders from indexing certain piece of content on your Web site.
• Depending on Web hosting, you may not have ability to create and add robots.txt file. In this case meta robots tag is the only method available to control the behavior of search engines.
• It is used on pages which you may not want to index, or which you wish only certain SE’s to index.
• It is a powerful way to prevent the search engines from indexing duplicate content, which is a very important thing to avoid.
For a more thorough discourse on the robots meta tag, see robots tag description