Large jobs fail for predictable reasons. Plan for them at the start and the job finishes; treat them as surprises and you will spend more time debugging than collecting.

Respect the rate

Start slow, measure the error rate, then increase carefully. A steady hundred pages per minute finishes the job. A burst that triggers a block in four minutes finishes nothing.

If the error rate rises above a few percent, slow down before you change anything else. Throttling correctly is usually the fastest way to finish sooner.

Rotate responsibly

  • Rotate user agents rarely and only when it is reasonable to do so
  • Use residential proxies only when the target genuinely requires them
  • Keep one session per logical user journey rather than per request
  • Preserve cookies within a session so the site sees consistent behaviour

Aggressive rotation often makes things worse. A site that sees a different identity on every request treats the traffic as suspicious far sooner than one seeing a consistent visitor.

Handle failures properly

  • Retry with exponential backoff instead of immediate re-requests
  • Distinguish a block from a genuine not found
  • Save progress continuously so the job resumes rather than restarts
  • Log the response code and body for every failure

Verify the output, not just the exit code

Row counts per page, completeness of required fields, and a spot check against the live page. A scrape that finishes with silent gaps is a failed scrape, and the failure is usually discovered only when the data is used.

Structure the job to be resumable

Write each page result as soon as it is parsed instead of holding everything in memory. Long jobs will be interrupted eventually, and a design that resumes from the last successful page costs almost nothing to build.

Know when to stop

How this looks in practice

A job to collect fifty thousand records starts at a cautious ten pages per minute. The error rate stays near zero, so the throttle is raised in small steps until failures appear at roughly one percent. It is then reduced slightly and left alone. The run finishes overnight with progress written to disk after every page.

Partway through, the site returns a small number of rate-limit responses. The script backs off, waits, retries and continues. No identity changes are needed because the pattern was steady from the beginning. The output is complete, and the log explains every gap.

That log matters later. When somebody questions a missing record, the answer is a timestamp and a response code rather than a shrug.

A short checklist

  • Start slow and tune upward from real error data
  • Keep one session per journey and preserve cookies
  • Write results as they are parsed, never at the end
  • Stop when a site clearly refuses automated access

Pair these habits with the rules in how to scrape a website ethically and legally, then clean the output using turning scraped data into a clean file.

If a site is clearly and deliberately preventing automated access, stop and look for an official API or an alternative source. Persisting against a determined block wastes budget and creates risk that the resulting data is not worth.