Skip to content
GET /tools/robots-txt-tester 200 OK
<scraping.pro/>
Request a quote
scraping.pro
Get in touch
  1. Home
  2. Tools
  3. robots.txt tester
Runs in your browserSEO & publishing

robots.txt tester

Paste the file, enter paths and bots, and see which rule decides.

Runs locally — nothing is uploaded
Result
Nothing yet — enter data or press Example.

How to use

  1. 1Paste the robots.txt content.
  2. 2Enter the paths and the bots you care about.
  3. 3Read the verdict and the rule that matched.

The most specific (longest) matching rule. If Allow and Disallow are equally long, Allow wins.

No, it only tests the text you paste. Use the live robots checker to fetch a site’s file.

Related tools

CSS selector testerTest selectors for scraping XPath testerXPath on HTML and XML curl convertercurl to Python, Node.js, PHP, Go, wget HTTP headers parserRaw headers to table, JSON or code Cookie parserSet-Cookie attributes and security flags HTML table to CSVTables from HTML to CSV

Need data from a website?

Send the URL and fields — we reply with a free sample and a quote.

Get in touchServices