Hello friends, Today i am going to write about how to add a robots.txt and sitemap.xml into website
First of all its important to know what is Robots.txt and Sitemap.xml file. I explain a brief introduction here for more information about them search google.
Robots.txt
Each search engine have its own crawlers like robots, scooters etc. When these crawlers goes to any site first of all they check the robots.txt file and then crawl the whole site. According to robots.txt file permission they crawl a website. For example if i restrict the google crawler to contact us page then my contact us page is not crawled. This is an idea about Robots and why we use the Robots.txt file. Here is the following code that is used for robots.txt file.
user-agent:*
Disallow:
First of all its important to know what is Robots.txt and Sitemap.xml file. I explain a brief introduction here for more information about them search google.
Robots.txt
Each search engine have its own crawlers like robots, scooters etc. When these crawlers goes to any site first of all they check the robots.txt file and then crawl the whole site. According to robots.txt file permission they crawl a website. For example if i restrict the google crawler to contact us page then my contact us page is not crawled. This is an idea about Robots and why we use the Robots.txt file. Here is the following code that is used for robots.txt file.
user-agent:*
Disallow: