What is robots.txt?
A small file on your site that tells search engine crawlers which areas they may or may not visit.
The plain definition
Robots.txt is a plain text file at the top level of your site, at an address like example.com/robots.txt. It gives instructions to crawlers about which parts of the site they should not visit. Well-behaved crawlers follow it, but it is a request, not a lock.
An example
A developer adds a line that blocks the whole site while it is being built, and forgets to remove it at launch. Search engines then stay away and the new site never appears. Another common use is blocking admin or internal search pages that add nothing.
Why it matters for a local business
Check your robots.txt after any launch or redesign, and be careful: one wrong line can hide everything. It does not keep a page private, because a blocked page can still be listed if others link to it. Use a noindex setting or a login for pages that must stay out of results.
Check it for free
See where you stand with the free website grader. If you would rather have it handled, our Websites service covers this as part of the monthly work.
Related terms
Part of the SEO glossary. New to all this? Start with the beginner’s SEO guide.
Last updated . Review by .