Scraping public information is common practice, and it is also easy to do badly. The difference between a legitimate data project and a legal problem usually comes down to four questions asked before any code is written.

Read three things first

  • The robots file - it states what the site owner expects from automated access
  • The terms of service — look specifically for clauses about automated collection
  • The nature of the data — is any of it personal information?

If any of these clearly forbids the activity, stop. There is almost always an alternative source.

Be a good citizen

  • Identify yourself honestly with a descriptive user agent
  • Limit request rate so the target site is not affected
  • Cache pages and avoid re-fetching what you already have
  • Scrape during off-peak hours where possible
  • Stop immediately if you receive a clear request to do so

Most sites tolerate reasonable automated access; almost none tolerate volume that degrades their service.

Personal data needs extra care

If the data includes names, email addresses or phone numbers, a legal basis is required in many jurisdictions. Keep the purpose specific and documented, store the data securely, retain it only as long as needed, and honour deletion requests promptly.

For B2B contact data, legitimate interest is often the basis used in Europe, but that comes with obligations: you must identify yourself, explain how the data was obtained if asked, and provide a working opt out.

When to use an official API instead

If the site offers an API, use it. It is faster, more stable and explicitly permitted. The cost of an API subscription is usually far lower than the engineering time needed to keep a scraper working against a site that is actively trying to prevent you.

Practical safeguards

  • Log exactly what you collect and when
  • Keep a clear business purpose in writing
  • Never scrape behind a login you were not given
  • Document your rate limits and user agent

Common mistakes

The most frequent error is assuming that public visibility equals permission to collect at any rate. Information can be public and the collection still unwelcome if it overwhelms a small server or gathers personal data without a stated purpose.

  • Ignoring the robots file because technically it is not code enforcement
  • Harvesting emails without a documented reason
  • Using a fake user agent to look like a search engine
  • Keeping personal data indefinitely because deletion feels like work

A short checklist

  • Read the robots file and the terms before writing code
  • Log your purpose and the fields you intend to keep
  • Set a rate you can explain to the site owner
  • Store personal data securely and delete it on schedule

Keep that record simple and current, so a client or regulator can see how a dataset was assembled without a long investigation.

FAQ

Is scraping public data legal? In many places it is permitted, but the answer depends on the data, the terms and the jurisdiction. Ask before assuming.

Do I need permission? For small, polite access to public pages, usually not. For personal data or high volume, document your basis and be ready to explain it.

To keep request patterns reasonable, read scraping large lists without getting blocked, and for a worked example see scraping Google Maps business data.

Good practice is also good business. Clients increasingly ask how their data was obtained, and a documented, responsible process is a straightforward answer rather than an awkward one.