DocsAccessrobots.txtBuild
Set up robots.txt
The structure a crawl-policy file needs, and the ordering mistakes that silently disable a rule.
- 1Your site
- 2What you have
- 3A few answers
- 4Your steps
Which site is this for?
We read what you already publish so most of the next step is filled in for you. Nothing is stored, and there is nothing to sign up for.
Reading
Only the documents this builder needs, fetched once, exactly as any agent would.
One question
Here is how
Everything below is generated from the same fixtures and rules the scanner checks against, plus what we just read from your site.