A crawl can be constrained by your computer before it is constrained by the website. Large URL sets, rendered pages, and retained response data all use local resources.
Plan storage before starting a long crawl. An export created at the end is useful for analysis, but it is not the same as a saved crawl that can be reopened with its relationships and configuration.
Understand memory and database storage
Screaming Frog supports memory and database storage approaches. Memory storage keeps more working crawl data in RAM. Database storage uses disk-backed storage and can be more suitable for larger datasets, while still needing memory and processing resources.
Database mode does not mean the crawl uses no RAM. It also does not mean that any slow external drive will perform well. Available space and disk speed matter.
Consult the publisher’s large-website crawling guidance for current configuration recommendations. Match allocation to the installed application and operating environment.
Estimate the data you need to retain
A basic response-and-metadata crawl has different requirements from rendered crawling with stored HTML or screenshots. Enable what the investigation needs, and record those choices.
Check free space before starting and allow room for the saved crawl, exports, and working analysis files. A nearly full disk can affect other applications as well as the crawler.
Avoid giving the crawler so much memory that the rest of the system becomes unstable. Close unnecessary workloads and monitor the machine during larger jobs.
Save the project before clearing results
Keep a dated crawl with a meaningful name. Include the site or section and the purpose, such as shop-category-audit-2026-09-20. Store the input list and a short settings note beside it.
A screenshot of the overview cannot replace the underlying dataset. If a developer asks for the source of an unexpected URL later, the saved crawl can preserve relationships that a summary export omitted.
Export for a specific question
CSV and spreadsheet exports are useful for filtering, grouping, and sharing. Select the fields needed for the finding: source and destination for a broken link, canonical values for a duplicate-version investigation, or extracted prices for a template check.
Google Sheets can be convenient for collaborative review, but large exports may need filtering or a database workflow. Choose the analysis environment based on the dataset rather than forcing the whole crawl into one sheet.
Keep provenance and clean up deliberately
Retain crawl date, tool version, mode, user agent, rendering, scope, and rate with the results. These settings explain why two crawls may differ.
Remove redundant working copies after confirming that the required saved crawl, original exports, and final findings are retained. Do not delete the only recoverable evidence simply because the issue has been assigned.
My existing large-scale log analysis with Python and MySQL covers a different dataset: server requests rather than a local crawl. My technical SEO consulting service and SEO automation service can help define storage and repeatable analysis for each source separately.