Modern pages often arrive as an empty shell. The content is injected by JavaScript after the page loads, which is why a simple request returns nothing useful and why so many first scraping attempts fail.

There are three ways to solve this, and one of them is almost always better than the others.

Option one: find the underlying API

Open the browser network tab, filter for fetch and XHR requests, and look for a JSON response that contains the data you want. In a large share of cases, the page is simply rendering data it fetched from an endpoint you can call directly.

This is the best option by a wide margin: faster, lighter, less fragile and far easier to maintain. Always check here first.

Option two: browser automation

Use Selenium or Playwright, wait for the specific element rather than a fixed delay, then extract. It is slower and heavier, but it works when no API exists or when interaction is required.

  • Wait for elements, never for time
  • Block images, fonts and analytics for speed
  • Reuse one browser session across pages
  • Capture a screenshot when something fails, to debug later

Option three: render once, parse separately

For very large jobs, run a headless browser cluster that saves the fully rendered HTML, then parse the saved files in a separate step. This separates rendering from parsing, which makes retries cheap and lets you re-parse with a fixed selector without re-fetching anything.

How to choose

  1. Is there a JSON endpoint? Use it.
  2. Is the data in the initial HTML? Use a simple request.
  3. Does the page need interaction or heavy JavaScript? Use a browser.
  4. Is the volume huge and the markup changing? Render once, parse separately.

Debugging tips

Disable JavaScript in the browser and reload. If the data disappears, the page is client-rendered. If it is still there, you never needed a browser in the first place — and that single check saves hours.

Also compare the request a real browser sends with the one your script sends. Differences in headers, cookies or query parameters are the most common cause of an empty response.