These meta tags which direct the indexing and crawling of a page has to be specified in the header section of HTML code.
noindex, follow
- To avoid indexing a page but allow crawling the following meta tag has to be specified in the page that is crawled:
<meta name="robots" content="noindex, follow">
index, nofollow
- To avoid crawling but allow indexing the following meta tag has to be specified:
<meta name="robots" content="index, nofollow">
noindex, nofollow
- To avoid both indexing and crawling the following meta tag has to be provided:
<meta name="robots" content="noindex, nofollow">
index, follow
- To index as well as crawl either you can avoid giving the meta robots or specify the same as shown:
<meta name="robots" content="index, follow">
To learn about robots meta tags for customized indexing read: Using Robots meta tag in HTTP Collection
Comments
0 comments
Please sign in to leave a comment.