How robots.txt allow/disallow matching works
Crawlers pick the most specific matching Allow or Disallow rule for their user-agent group. This tester applies a practical subset of common matching rules so you can spot accidental blocks before deploy.
robots.txt Tester
Test how a robots.txt file treats a path for a given user-agent. See the matched agent group and a short reason for allow or disallow.
How it works
- 1
Fill in the form fields the tool shows.
- 2
Change modes or toggles if available.
- 3
Copy the generated output when it looks right (often updates live).
Why use this tool
- Runs entirely in your browser — nothing is uploaded to a server.
- 100% free, no sign-up, no limits.
- Works in any modern browser — no app install, no extension, no account.
How to use robots.txt Tester
Complete the fields in robots.txt Tester. Output is produced in your browser from those inputs — copy it when ready.
Tips for better results
Empty optional fields if you want a minimal result. Invalid combinations should surface an error message.
Privacy and offline-friendly processing
Whenever possible, robots.txt Tester processes your data locally with JavaScript in this page. That means drafts, secrets, and unfinished work stay on your device by default. Prefer this workflow when you do not want to upload sensitive text, tokens, passwords, or files to a third-party service.
Frequently asked questions
Does this fetch my live robots.txt?
No. Paste the file contents yourself—nothing is fetched from your site.
Is this a full Googlebot simulator?
It covers common Allow/Disallow patterns with wildcards. Edge cases in crawler implementations may still differ.
Is my data uploaded when I use this tool?
For most FlyUtils tools, no. Processing happens in your browser with JavaScript. If a tool ever needs a server, that will be stated clearly on the page.
Is robots.txt Tester free to use?
Yes. robots.txt Tester is free, with no sign-up and no artificial usage caps for normal personal and work use.
Does robots.txt Tester work on mobile?
Yes. The page is mobile-first, so you can paste input, tweak options, and copy results from a phone or tablet.