Crawling and indexing
Crawling is search engines visiting and reading your web pages. Indexing is storing those pages in the search engine’s database, so they can appear in the search results.
- 3 min read
What are crawling and indexing?
Search engines use programs that scan the web, known as crawlers or bots. Google’s crawler is called Googlebot. It follows links from page to page, reads the content and the code, and picks up new or changed pages. That is called crawling.
After crawling, Google decides whether a page is added to the index, the enormous database that search results are drawn from. That is called indexing. Only an indexed page can appear in Google.
So crawling does not automatically mean indexing. Google can read a page and still decide not to include it, for example because the content is thin or seen as a copy. AI systems have their own crawlers too, such as GPTBot, PerplexityBot and Anthropic’s crawler. Read more about that in llms.txt.
Why it matters for your business
A page that is not indexed does not exist for Google. You can have the best page about roof renovation in your region, but if it is not in the index, it brings in no enquiries at all through search engines.
That sounds like an exception, but it happens often. After a new website, a move to another domain or a plugin update, pages regularly end up on noindex or blocked by accident. Without checking, you only notice when enquiries drop.
The same goes for AI search. If your robots.txt shuts out AI crawlers, you cannot be mentioned in their answers either.
How it works
Google finds your pages in three ways: through links from other sites, through links within your own site and through your XML sitemap, a list of all the pages you want indexed. The robots.txt file determines which parts of your site crawlers may visit. With a noindex tag on a page, you say it does not belong in the index.
Google then assesses the page. Is the content useful? Is it not a copy of another page? Does the canonical URL point to itself? Is the page technically accessible and fast enough?
An example. Say an installation company has a new website built on a test environment. During the build, the WordPress option that asks search engines not to index the site is switched on. At go-live, that option is forgotten. On top of that, the old addresses are not redirected to the new ones. A few weeks later, the number of enquiries via Google has dropped sharply. Search Console shows that almost all pages are on noindex and that the old addresses return an error.
After switching off the noindex, setting up redirects from old to new and submitting the sitemap, the pages are crawled and indexed again. Recovery takes time that could have been avoided with a check at go-live.
Common mistakes
The best-known error is a forgotten noindex after go-live, as in the example. It is one tick box, but the effect is big.
A second error is a robots.txt that blocks too much. Sometimes a rule accidentally blocks the whole site, or the folders with images and scripts that Google needs to read the page properly.
Also often seen: pages that no internal link points to. Such orphan pages are hard to find. And a sitemap that contains old, redirected or non-existent addresses gives Google the wrong signals.
Finally, many business owners expect a new page to be in Google straight away. Indexing can take days to weeks, especially on a new or small site. Requesting indexing again and again does not speed it up. It is better to make sure the page is well linked and genuinely adds something.
How to approach it
Connect your site to Google Search Console. In the indexing report you see which pages are included and which are not, with the reason. Check this after every major change.
Submit your XML sitemap via Search Console and make sure it only contains pages you really want indexed. Use the URL Inspection function to test a new or important page and request indexing.
Check your robots.txt. Do not block important parts, and decide deliberately whether to allow AI crawlers. For most trade businesses, allowing them is sensible, because it means you can appear in AI answers.
Make sure every important page is reachable through internal links. A new website always comes with a redirect plan and a noindex check at go-live. That is a fixed part of technical SEO.
Frequently asked questions
Don't see your question here? Call, WhatsApp or email us. You'll get an answer within one working day.
How do I know if my page is in Google?
The most reliable way is Google Search Console. With URL Inspection you can see for each page whether it is indexed and, if not, why. A quick check is also possible by searching Google for site: followed by the page address.
How long does it take for a new page to be indexed?
It varies from a few days to several weeks. A site that is updated often and well linked gets crawled faster. You can speed it up by including the page in your sitemap and requesting indexing in Search Console.
Should I allow AI crawlers on my site?
For most trade businesses, that is sensible. If you block crawlers such as GPTBot, PerplexityBot or Anthropic’s crawler, AI assistants cannot read your content and are less likely to mention you in their answers. Check in your robots.txt that there is no block you did not choose deliberately.
What is Google Ads and how does it work?
Google Ads explained: how the auction system works, which campaign types there are and how you decide what a click is allowed to…
Local SEO: get more from your local visibility
Local SEO makes sure customers near you find you on Google and Google Maps.
What does a marketing agency do?
What exactly does a marketing agency do, and when do you need one?
Ready for more enquiries?
Choose where your growth should come from. A specialist picks it up within one working day.
Marketingkennersonline · reply within one working day