A file at the root of a website that provides crawl instructions to compliant robots.
A practical example
A site can disallow a crawler from requesting a particular URL path.
What to watch for
Disallowing crawling does not reliably prevent indexing of a discovered URL. To remove a crawlable page from indexing, use a supported noindex directive; use authentication for private content.
Related reading
Explore Indexing and Metatag. Try our noindex checker for a practical next step. See how this fits into Technical SEO.