Scraping public information is common practice, and it is also easy to do badly. The difference between a legitimate data project and a legal problem usually comes down to four questions asked before any code is written.

Read three things first

  • The robots file - it states what the site owner expects from automated access
  • The terms of service — look specifically for clauses about automated collection
  • The nature of the data — is any of it personal information?

If any of these clearly forbids the activity, stop. There is almost always an alternative source.

Be a good citizen

  • Identify yourself honestly with a descriptive user agent
  • Limit request rate so the target site is not affected
  • Cache pages and avoid re-fetching what you already have
  • Scrape during off-peak hours where possible
  • Stop immediately if you receive a clear request to do so

Most sites tolerate reasonable automated access; almost none tolerate volume that degrades their service.

Personal data needs extra care

If the data includes names, email addresses or phone numbers, a legal basis is required in many jurisdictions. Keep the purpose specific and documented, store the data securely, retain it only as long as needed, and honour deletion requests promptly.

For B2B contact data, legitimate interest is often the basis used in Europe, but that comes with obligations: you must identify yourself, explain how the data was obtained if asked, and provide a working opt out.

When to use an official API instead

If the site offers an API, use it. It is faster, more stable and explicitly permitted. The cost of an API subscription is usually far lower than the engineering time needed to keep a scraper working against a site that is actively trying to prevent you.

Practical safeguards

  • Log exactly what you collect and when
  • Keep a clear business purpose in writing
  • Never scrape behind a login you were not given
  • Document your rate limits and user agent

Good practice is also good business. Clients increasingly ask how their data was obtained, and a documented, responsible process is a straightforward answer rather than an awkward one.