How to create a Robot.txt | what is a Robots.txt | Robots exclusion standard – Part 8


We should use robots.txt file to restrict any search engine for crawling. In this video WsCube Tech explain you how to create robots.txt.

The Robot Exclusion Standard = Robots Exclusion Protocol = robots.txt Protocol

1. You need a robots.txt file only if your site includes content that you don’t want search engines to index like Your site Admin Panel.

2. User-agent: *Disallow: /Admin Panel/

3. less then bracket meta name=”robots” content=” noindex,nofollow“ greater then bracket

4. less then bracket meta name=”googlebot“ content=”noindex” greater then bracket

5. X-Robots-Tag: noindex

Link –

Learn Digital Marketing : Download APP

This app is a complete guide for learning digital marketing step by step.

Follow Us On :
Facebook :
Linked in :
Google+ :
YouTube :
Twitter :

Leave a Reply

Your email address will not be published. Required fields are marked *

19 − six =