Generate and test a robots.txt.
Add per-User-agent groups, attach Allow and
Disallow rules (with wildcards and $ end-anchors),
point at your sitemaps, and preview the resulting file live. Then paste a URL
to see which rule the matcher picks and whether the page is allowed or blocked.
Editor
Each User-agent group is matched independently. A bot will use
the most specific (longest Allow or Disallow) rule
from the group that names it, falling back to the * group when
nothing else applies. Paths are case-insensitive.
Related reading
Where robots.txt sits next to the rest of the SEO and security baseline.
Meta Tag & OpenGraph Inspector
Audit the <head> metadata that controls search snippets and social previews.
sitemap.xml
Pair this robots.txt with a sitemap the crawlers can actually fetch.
Security Headers Checker
Confirm X-Robots-Tag and friends match what your robots.txt allows.
CSP Generator
Generate a strict Content-Security-Policy for the same origin.
Security headers best practices
How robots.txt, meta robots, and X-Robots-Tag fit together.
CSP not working
If the policy and the robots.txt disagree, start here.