TL;DR: Screaming Frog SEO Spider = most powerful crawling tool for SEO. Free up to 500 URLs, paid £149/year for unlimited. Essential for audits, migrations, technical SEO.
Main Uses
1. Site crawl
Crawl entire site. Get all URLs with metadata.
2. Broken links
Find 404s, 301s, 500s.
3. Duplicate content
Find near-duplicate pages.
4. Meta tag audit
Missing, duplicate, too long titles/descriptions.
5. Internal linking
Pages with too few/many internal links.
6. Schema validation
Find missing or broken schema.
7. Redirects
Redirect chains, loops.
Setup
- Download from screamingfrog.co.uk.
- Install.
- Free mode or license.
- Enter domain in URL bar.
- Start crawl.
Configuration
- User-agent. select Googlebot for realistic crawl.
- JS rendering. enable for SPA sites.
- Speed. limit to avoid overwhelming server.
- Include/exclude. specific URLs.
Reports to Run
- Response Codes. 404s, 500s.
- Page Titles. missing, duplicate.
- Meta Description.
- H1. missing, duplicate.
- Images. missing alt, oversized.
- Canonical. chains, issues.
- Hreflang. misconfigured.
- Schema. extracted.
Advanced
- Custom extraction. scrape specific content.
- API integration. Ahrefs, Semrush.
- Scheduled crawls. automation.
- Compare crawls. changes over time.
רוצה עזרה ממוקדת ל-SEO באתר שלך?
אני שמוליק דורינבאום — יועץ SEO ב-20+ שנות ניסיון. אם המאמר הזה נתן לך כיוון ואתה רוצה ליישם על האתר שלך, יש לי קורס קידום אתרים חינמי מלא, או שאפשר לקפוץ לשיחת ייעוץ אישית.