Large jobs fail for predictable reasons. Plan for them at the start and the job finishes; treat them as surprises and you will spend more time debugging than collecting.
Respect the rate
Start slow, measure the error rate, then increase carefully. A steady hundred pages per minute finishes the job. A burst that triggers a block in four minutes finishes nothing.
If the error rate rises above a few percent, slow down before you change anything else. Throttling correctly is usually the fastest way to finish sooner.
Rotate responsibly
- Rotate user agents rarely and only when it is reasonable to do so
- Use residential proxies only when the target genuinely requires them
- Keep one session per logical user journey rather than per request
- Preserve cookies within a session so the site sees consistent behaviour
Aggressive rotation often makes things worse. A site that sees a different identity on every request treats the traffic as suspicious far sooner than one seeing a consistent visitor.
Handle failures properly
- Retry with exponential backoff instead of immediate re-requests
- Distinguish a block from a genuine not found
- Save progress continuously so the job resumes rather than restarts
- Log the response code and body for every failure
Verify the output, not just the exit code
Row counts per page, completeness of required fields, and a spot check against the live page. A scrape that finishes with silent gaps is a failed scrape, and the failure is usually discovered only when the data is used.
Structure the job to be resumable
Write each page result as soon as it is parsed instead of holding everything in memory. Long jobs will be interrupted eventually, and a design that resumes from the last successful page costs almost nothing to build.
Know when to stop
If a site is clearly and deliberately preventing automated access, stop and look for an official API or an alternative source. Persisting against a determined block wastes budget and creates risk that the resulting data is not worth.