What Is Robots.txt?

Discussion in 'SEO and Marketing' started by evargro, Apr 15, 2019 at 12:18 AM.

Share This Page

  1. evargro

    evargro New Member

    Joined:
    Jan 21, 2019
    Messages:
    28
    Likes Received:
    0
    What Is Robots.txt?
     
  2. sinelogixtech

    sinelogixtech Senior Member

    Joined:
    Mar 21, 2017
    Messages:
    293
    Likes Received:
    5
    The robots.txt file, also known as the robots exclusion protocol or standard, is atext file that tells web robots (most often search engines) which pages on your site to crawl. It also tells web robots which pages not to crawl.The slash after “Disallow” tells the robot to not visit any pages on the site.