Free robots.txt Checker
Inspect the robots.txt file discovered from your URL's origin, see which crawler groups and rules apply, and verify referenced sitemaps.
What is robots.txt?
The Robots.txt Checker reads the site-level file from the URL's origin, parses crawler-specific groups, evaluates wildcard rules against discovered URLs, and checks declared sitemap references.
Why it matters
A misconfigured robots.txt file can accidentally block important pages or let bots crawl low-value areas.
Common problems
- Scanned or discovered URLs blocked by a rule
- Whole-site Disallow rule
- Malformed crawler groups or unsupported directives
- Sitemap references unavailable or using HTTP
How to fix them
Review the exact crawler group and matching rule first. Change only restrictions that conflict with the site's intended crawl policy.
Example
Example: /admin/ is blocked but /products/ is also unintentionally blocked → search engines cannot discover the main catalog pages.
FAQ
Does robots.txt control ranking?
It controls crawling, not direct ranking. A page can still be indexed if it is linked and accessible even if caching or crawl rules are imperfect.
Should I block CSS or JavaScript?
Usually no. Blocking resources can prevent rendering and indexing issues if bots cannot evaluate the page properly.
Is it free?
Yes.