Separate names with a comma.
Discussion in 'SEO and Marketing' started by evargro, Apr 15, 2019 at 12:18 AM.
What Is Robots.txt?
The robots.txt file, also known as the robots exclusion protocol or standard, is atext file that tells web robots (most often search engines) which pages on your site to crawl. It also tells web robots which pages not to crawl.The slash after “Disallow” tells the robot to not visit any pages on the site.