Auditing a website by clicking through it by hand collapses at about ten pages. A crawler does not collapse at all: mastering how to use Screaming Frog for SEO audits means pointing an industrial-grade spider at any domain and extracting every broken link, bad redirect and rogue directive in minutes instead of weeks.
Screaming Frog SEO Spider is a desktop application that crawls the way search bots do — following links, recording status codes, metadata, canonicals and robot directives — then lets you filter, sort and export everything found. The free licence stops at 500 URLs per crawl, which covers smaller sites completely and gives larger ones an honest sample to start from.
Quick Answer: Install the spider, confirm robots.txt respect and the right user agent, crawl the site, then work the Response Codes and Directives tabs before exporting filtered spreadsheets that convert findings directly into developer fix tickets.
Phase One: Configure Before You Crawl
- Enter the homepage as your start URL and press Start — defaults suit most first audits.
- Leave robots.txt respect enabled unless you have explicit permission to audit a staging environment that blocks bots.
- For JavaScript-dependent sites, switch Configuration so the spider renders pages; expect noticeably slower runs.
- Exclude faceted-navigation and parameter patterns upfront so the crawl isn't drowned in duplicate URL variants.
Saving this configuration as a reusable profile keeps every future audit measured on identical terms.
Phase Two: Work the Response Codes Report
Nothing else matters until transport is clean. Within Response Codes, the Client Error (4xx) filter lists dead internal links, Server Error (5xx) catches pages failing intermittently, and Redirection hides the sneaky stuff: chains. A redirect pointing at another redirect burns crawl budget and dilutes signals along the way — export the chain, then repoint the first hop straight at the final destination.
Also glance at the Success (200) count against expectations. Wildly inflated totals usually signal index bloat from pagination or filtered navigation rather than genuine content growth.
Phase Three: Audit the Directives Tab
This tab collects everything pages tell crawlers to do — and where instructions contradict each other:
- Meta robots and X-Robots-Tag: a stray noindex on a money page is the single most expensive accident in SEO, and this is where you find it.
- Canonicals: confirm each important URL self-canonicalises; flag pages pointing their canonical elsewhere despite being the intended original.
- Nofollow directives and hreflang: round out the picture on larger, international builds.
Phase Four: Sweep On-Page Elements in Bulk
The Page Titles, Meta Description and H1 tabs expose duplicates, missing fields and truncation at a glance. Filter for duplicates first — they usually indicate templating bugs rather than careless editors, and fixing one template repairs hundreds of URLs at once. The Images tab catches missing alt text en masse, and Custom Extraction (XPath, CSS selectors or regex) mines anything pattern-addressable: prices, author names, schema presence, outdated dates.
Phase Five: Export Findings Into Fixable Artifacts
Every filter carries an Export button producing spreadsheets developers can act on without access to the tool. Bulk Export grabs whole report families simultaneously; paid licences add database storage so successive crawls can be compared row by row, plus scheduled runs that audit continuously in the background.
| Report or Tab | Problem It Surfaces | Typical Remedy |
|---|---|---|
| Response Codes | 404s, 500s, redirect chains | Restore pages; flatten chains to one hop |
| Directives | Stray noindex, conflicting canonicals | Delete accidents; enforce self-referencing canonicals |
| Titles, metas, H1s | Duplicates, truncation, absence | Fix templates before editing individual copy |
| Custom Extraction | Missing schema, stale prices, thin sections | Targeted regex sweeps with exports |
Turning Audits Into a Habit
Single audits find fires; repeated ones prevent them. Even on the free tier, recrawling after every major release and diffing exports manually keeps regressions visible before Google notices them. Agencies and larger sites justify the licence precisely for that continuity — saved profiles, crawl comparison and scheduling compound in value over time.
Key Takeaways
- Configure exclusions and rendering before crawling, or you will re-run everything.
- Clear Response Codes first — audits built on broken transport mislead.
- The Directives tab catches the costly silent killers: stray noindex, wrong canonicals.
- Exports, not screenshots, turn crawl data into completed fixes.