Modern pages often arrive as an empty shell. The content is injected by JavaScript after the page loads, which is why a simple request returns nothing useful and why so many first scraping attempts fail.

There are three ways to solve this, and one of them is almost always better than the others.

Option one: find the underlying API

Open the browser network tab, filter for fetch and XHR requests, and look for a JSON response that contains the data you want. In a large share of cases, the page is simply rendering data it fetched from an endpoint you can call directly.

This is the best option by a wide margin: faster, lighter, less fragile and far easier to maintain. Always check here first.

Option two: browser automation

Use Selenium or Playwright, wait for the specific element rather than a fixed delay, then extract. It is slower and heavier, but it works when no API exists or when interaction is required.

  • Wait for elements, never for time
  • Block images, fonts and analytics for speed
  • Reuse one browser session across pages
  • Capture a screenshot when something fails, to debug later

Option three: render once, parse separately

For very large jobs, run a headless browser cluster that saves the fully rendered HTML, then parse the saved files in a separate step. This separates rendering from parsing, which makes retries cheap and lets you re-parse with a fixed selector without re-fetching anything.

How to choose

  1. Is there a JSON endpoint? Use it.
  2. Is the data in the initial HTML? Use a simple request.
  3. Does the page need interaction or heavy JavaScript? Use a browser.
  4. Is the volume huge and the markup changing? Render once, parse separately.

Debugging tips

Disable JavaScript in the browser and reload. If the data disappears, the page is client-rendered. If it is still there, you never needed a browser in the first place — and that single check saves hours.

A short checklist

  • Check the network tab for a JSON endpoint first
  • Disable JavaScript to see what the server really returns
  • Wait for a specific element, never a fixed delay
  • Reuse one browser session and capture screenshots on failure

Common mistakes

The costliest habit is reaching for a browser before checking for an endpoint. Rendering every page multiplies runtime, memory and failure points, when a single request would often return the same JSON.

  • Sleeping for a fixed number of seconds instead of waiting for the element
  • Starting a fresh browser session for every page, which is slow and suspicious
  • Parsing from a screenshot or from text that shifts between runs
  • Assuming a blank body means the data does not exist

Most of these come from solving the visible problem instead of the underlying one.

What to measure

Track page load time, rows extracted per page and the percentage of pages that need a retry. When retries climb, the wait condition is usually wrong. Slower, steadier browser work also benefits from the throttling habits in scraping large lists without getting blocked. If you are still choosing a tool, start with web scraping with Python.

Also compare the request a real browser sends with the one your script sends. Differences in headers, cookies or query parameters are the most common cause of an empty response.