Crawling
The process by which search engines use automated bots to discover and read pages across the web.
Crawling is how search engines find your content. Automated bots (crawlers) follow links from page to page, reading what they find so it can later be added to the search index.
What helps crawling
Clear internal links, a sitemap, and a clean site structure help crawlers find all your important pages. A robots.txt file can tell crawlers which areas to skip.
Why it matters
If a crawler can’t reach a page, that page can’t appear in search at all. Crawling is the first step before anything else in SEO works.
Related Terms
- SEO Basics
301 Redirect
A permanent redirect that sends visitors and search engines from an old URL to a new one, passing along most of the old page's ranking value.
- SEO Basics
Alt Text
A written description of an image in a page's HTML, used by screen readers and search engines to understand what the image shows.
- SEO Basics
Anchor Text
The visible, clickable words of a hyperlink, which give readers and search engines a clue about the linked page's topic.