A crawler can generate requests faster than a person clicking through a site. Increasing its speed can shorten a job, but it can also trigger rate limiting or put extra pressure on the application.
Agree an appropriate rate with the infrastructure team. The right setting depends on the application, current traffic, caching, and the work each request causes. There is no universal safe speed for every website.
Separate concurrency from request rate
Max Threads controls concurrent work. The URLs-per-second control, labelled Max URI/s in current documentation, caps the request rate. These are related limits, not numbers to multiply together.
If the allowed rate is 120 URL requests per minute, the simple conversion is:
120 requests ÷ 60 seconds = 2 requests per second
A two-per-second ceiling does not mean two requests per second for every thread. More threads may help sustain a rate when responses take time, but the global rate setting still constrains throughput.
The official speed configuration section documents the controls. Confirm the labels in your installed version.
Account for the actual server work
A URL request can trigger more than reading a static file. The application may run database queries, call services, or generate a page that is not cached. Rendered crawling can also require supporting-resource requests.
A nominal rate should therefore be evaluated against the requests and load it actually produces. Ask which monitoring view the infrastructure team will use and whether a quieter test window is available.
Do not assume a fast laptop means the website can accept more requests. Client capacity and origin capacity are different constraints.
Watch responses during the crawl
Monitor response times, 5xx failures, 429 rate-limit responses, and unexpected 403s. Compare with infrastructure metrics where available. A rising error count can mean the test conditions have changed and the dataset is becoming less representative.
If problems appear, pause or reduce the rate, record the time, and coordinate with the team. Retain the affected requests so you can distinguish site behaviour from load introduced by the crawl.
Keep local constraints separate
CPU, memory, disk performance, and rendering work can limit the crawler’s achieved speed. A low observed rate does not necessarily mean the server is slow. Check the local machine and storage configuration before changing server-side settings.
Conversely, reducing local bottlenecks can suddenly increase the load sent to the site. Recheck the agreed rate when changing hardware or crawl configuration.
Record the operational conditions
Save the configured limits, achieved rate, time window, rendering mode, and any access arrangements. If you repeat the crawl, compare the same page set under similar conditions before interpreting response-time differences.
A technical SEO audit needs reliable data without creating avoidable production errors. My technical SEO consulting service includes coordinating crawl conditions and connecting the results with server-side evidence when performance or access changes during testing.