Crawl & log analysis
/services/crawl-logs
- Problem
- What Googlebot actually did, not what a tool guessed
- Scope
- What Googlebot actually did, taken from your raw server logs rather than inferred from a crawling tool. On a large site the two rarely match, and the gap between them is usually where the traffic went. The crawl audit gives us the map of your site; on the Retainer and Embedded plans, monthly log analysis shows how Googlebot is really using it.
- Checks
- Full crawl and server log pipeline
- Crawl budget by template
- Faceted navigation control
- Parameter and duplicate handling
- Output
- Crawl data for every template, with status code, canonical and directives per URL
- Googlebot requests by template, with wasted requests counted per day (Retainer and Embedded)
- A per-parameter decision table your developers can implement and keep under version control
- Findings ranked by the number of indexable pages affected, each with its evidence