Skip to main content

InstrumentsRobots.txt Tester

Robots.txt Tester

Paste a robots.txt file, choose a user-agent and a path, and see whether that crawler may fetch the path, with the rule that decided it and any warnings.

All tools

Robots.txt Tester

This tester follows the longest-match rule: the longest matching pattern wins, and on a tie Allow wins. Other crawlers may differ. It checks one path at a time. It does not fetch the live robots.txt, so it makes no network calls.

One directive per line, such as User-agent: *, Disallow: /admin or Allow: /admin/public. Lines starting with # are comments. Rules that appear before any User-agent line are ignored, as crawlers do. Sitemap and Crawl-delay lines are read but not used.

Matched without regard to case against each User-agent line. Leave it as * to use the * group.

Starts with /. Matching is case-sensitive. Include the query string if it matters, for example /file.pdf?v=2.

Result

Allowed

* may fetch /.

No rule matched this path, so it is allowed by default.

FAQ

Which rule wins when Allow and Disallow both match a path?

This tester uses the longest-match rule: the longest matching pattern wins, and when two matching patterns are the same length, Allow wins. Other crawlers may choose differently.

Does this tool fetch my live robots.txt?

No. The text you paste is checked in your browser, one path at a time. Nothing is sent over the network.

Does a Disallow rule remove a page from search results?

No. Robots.txt controls crawling, not indexing. A blocked URL can still appear in results if other pages link to it. To keep a page out of results, use a noindex robots tag, and leave the page crawlable so the tag can be read.

  1. 01Fix Google Search Console Indexing in Next.jsTutorial
  2. 02Next.js App Router TutorialTutorial

Personalize your path

Choose your experience level once.

Next Step

Build fundamentals after each tool run so you can apply concepts with confidence.

Get new tool alerts

Receive updates when new tools are published.

RSS feed