What is crawling?
When a search engine’s program visits pages and follows links to find content. A page it cannot reach cannot be listed.
The plain definition
Crawling is how search engines discover pages. A program, often called a crawler or bot, visits a page, reads it, and follows the links on it to find more pages. Everything a search engine knows about the web starts with this process. If a page cannot be crawled, it cannot be added to the index.
An example
Your new “Water Heater Repair” page is not linked from anywhere on your site, and it is missing from your sitemap. The crawler never finds it, so it never shows in search. Linking to it from your services page and listing it in the sitemap solves the problem.
Why it matters for a local business
You help crawlers by keeping a clear structure, linking your pages to each other, fixing broken links, and not blocking important pages by mistake. A stray line in a robots.txt file or a “noindex” setting left on after a redesign is a common reason a good page goes missing.
Check it for free
See where you stand with the free website grader. If you would rather have it handled, our Websites service covers this as part of the monthly work.
Related terms
Part of the SEO glossary. New to all this? Start with the beginner’s SEO guide.
Last updated . Review by .