SiteOne Crawler

SiteOne Crawler is described on its page as a free Swiss-Army-knife for websites. It is a dependency-free binary that deep-crawls any site and produces an interactive HTML audit report with a 0-10 quality score across Performance, SEO, Security, Accessibility and Best Practices, a full offline copy (mirror) of a site, or clean Markdown for AI use. It is offered as a desktop app (GUI) and a command-line tool (about 200 options), with an interactive wizard, CI/CD quality gate, and local export server mentioned. Packages are listed for Homebrew, apt, dnf, zypper, apk and a Windows zip download. The page states it is open-source and free, and lists GitHub as the project home.

Key features

• Deep crawl of every asset
• Interactive HTML audit report with 0-10 quality score across five areas
• Fix-first panel ranking highest-impact issues
• Full offline clone (mirror) including assets, fonts, documents and 3rd-party resources
• Website to Markdown or llms.txt conversion, plus standalone HTML-to-Markdown and built-in viewer
• Optional headless Chromium (--browser) crawling of JavaScript and SPA sites
• Headful browser mode, per-page screenshots to GIF or MP4, and console/network diagnostics
• Bring-your-own-LLM AI assistant with OpenAI, Anthropic, Google Gemini, Ollama, vLLM, LM Studio or OpenAI-compatible endpoints, with --ai-dry-run
• CI/CD quality gate (--ci) with configurable thresholds, JUnit XML, GitHub annotations and exit code 10 on failure
• SEO and OpenGraph checks
• Security headers and cookies, SSL/TLS certificate checks
• Accessibility (WCAG) checks
• Performance and slow page checks
• Broken links, 404s and redirect checks
• Best practices, headings and caching header checks
• Content types and sizes checks
• Sitemap generator
• Stress and load testing
• Scheduled e-mail reports
• Local export server

Best suited for

• Website owners and maintainers running SEO and quality audits
• Teams wanting a CI/CD quality gate or post-deploy cache warming
• Archiving or disaster-recovery copies of websites as offline mirrors
• Preparing website content as Markdown for LLMs, AI models, llms.txt or documentation
• Developers crawling JavaScript and SPA sites with a headless browser

Pricing

Not stated on the website