Three tools, three different jobs. Choosing correctly at the start saves days of rework later, because migrating a Scrapy project into Selenium halfway through is far more expensive than choosing well on day one.
Requests plus BeautifulSoup
Best for simple pages where the data already exists in the delivered HTML. Fast to write, easy for a colleague to read, and perfectly adequate for a few hundred pages.
Its weakness is that it does no pagination, retries or concurrency on its own. Once a project needs those, it is time to move on.
Scrapy
Built for scale. Queues, retries, throttling, concurrency and item pipelines are all included, so you spend your time on the extraction logic rather than the plumbing.
- Thousands to millions of pages
- Multi-domain crawls
- Structured output through pipelines
- Resume and pause support for long jobs
Selenium or Playwright
Needed when the page builds itself with JavaScript, or when you must click, scroll or log in to reach the data. Slower and heavier, so use it only when the static request approach genuinely cannot see the content.
The decision table
- Static HTML, small volume — requests plus BeautifulSoup
- Static HTML, large volume — Scrapy
- JavaScript heavy or interaction required — Selenium or Playwright
- Data behind a login you have access to — browser automation with a saved session
- Hidden API behind the page — inspect the network tab first, then call the endpoint directly
That last option is the most underrated. Many modern sites load their own data from a JSON endpoint, which is faster, cleaner and far less fragile than rendering the page.
Non-negotiables in every scraper
- Retries with exponential backoff
- Polite delays and a clear user agent
- A check that the page actually loaded before parsing
- Progress saved so a long job can resume
- Row counts logged per page
That final point matters most. A scraper that silently returns half the rows is worse than one that fails loudly, because wrong data gets used for decisions before anyone notices.