Web scraping and browser automation

Some platforms have no API, or an API that omits the thing you need. Fetching their pages, or driving a browser, is then the integration. It is a legitimate engineering decision provided you treat it as the fragile, expensive and legally loaded option it is.

Senior questions here are less about selectors than about politeness, change detection, silent failure and the law: “every request returns 200 but the dataset keeps shrinking”, “re-crawl a million pages nightly without re-downloading the unchanged ones”, “is this legal?”.

The decision order

Work down this list and stop at the first one that exists:

1. Official API. Stable contract, versioned, supported.
2. Undocumented internal API. The endpoints the site’s own frontend calls. Faster and far more stable than driving a browser, but it can change without notice.
3. Fetch and parse HTML. No JavaScript needed.
4. Browser automation. Drive the real UI.

Step 2 is the one people skip. Open the network tab, watch what the page requests, and you frequently find a clean JSON endpoint behind the interface. Replaying it costs milliseconds where a browser costs seconds.

# What the page itself calls, found in the network tab.
r = await client.get(
    "https://app.example.com/api/v2/orders",
    headers={"authorization": bearer, "accept": "application/json"},
)

Which tool, once the answer is “not an API”

Scrapy is a crawling framework, not a fetching library. Over a loop around httpx it gives you a scheduler, per-domain concurrency and delay, retries, deduplication of seen URLs, an item pipeline, an HTTP cache and resumable crawls (JOBDIR). It does not render JavaScript; scrapy-playwright renders only the pages that need it, which matters because rendering every page wastes most of the budget.

The honest default: start with httpx plus selectolax or lxml. Most jobs are a few hundred pages behind a JSON endpoint found in step 2. Move to Scrapy when the crawl has breadth and needs politeness and resumability.

HTTP client defaults that bite crawlers

• requests has no default timeout: one stalled socket hangs the worker forever. Always pass timeout=.
• httpx defaults to a 5-second timeout and does not follow redirects unless follow_redirects=True; a crawl that silently stores 301 bodies is this.
• Encoding: when the Content-Type has no charset, httpx decodes as UTF-8 instead of guessing, which garbles windows-1251 or Shift_JIS pages. Pass default_encoding with a detector, or decode from r.content yourself.
• Parsers: lxml and parsel (XPath and CSS) and selectolax (CSS only, the fastest) are the production choices; BeautifulSoup is the friendliest API and runs on top of either html.parser or lxml. They disagree on broken HTML (an unclosed table), so test selectors against the parser you deploy.

Politeness: robots.txt and rate

robots.txt has been an IETF standard since RFC 9309 (September 2022), and the standard itself says it is not access control. What it specifies:

• The most specific (longest) matching rule wins, and Allow beats Disallow on a tie; * and $ are wildcards.
• A 4xx for robots.txt means you may crawl; a 5xx or an unreachable server means assume everything is disallowed.
• Parsers must handle at least 500 KiB.
• Crawl-delay is not in the standard. Bing honours it; Google ignores it. Rate-limit yourself regardless.

Python’s own parser only matches the standard from 3.14. The same rules on three interpreters:

User-agent: *
Disallow: /shop/
Allow: /shop/public/
Disallow: /*.pdf$

3.12.6   /shop/public/item    disallowed
3.12.6   /docs/a.pdf          allowed
3.13.5   /shop/public/item    disallowed
3.13.5   /docs/a.pdf          allowed
3.14.5   /shop/public/item    allowed
3.14.5   /docs/a.pdf          disallowed

Before 3.14, urllib.robotparser takes the first matching line and ignores wildcards, so it reports the opposite of what the site asked for in both cases. On older Pythons use protego (Scrapy’s parser), which implements the standard.

A bounded, polite fetch loop

import asyncio, random
from collections import defaultdict
from urllib.parse import urlsplit
import httpx

GLOBAL = asyncio.Semaphore(50)
PER_HOST = defaultdict(lambda: asyncio.Semaphore(2))   # in flight per host
UA = "acme-catalog-bot/1.0 (+https://acme.example/bot; bot@acme.example)"

async def fetch(client: httpx.AsyncClient, url: str, tries: int = 5):
    host = urlsplit(url).hostname
    for attempt in range(tries):
        async with GLOBAL, PER_HOST[host]:
            r = await client.get(url)
        if r.status_code not in (429, 503):
            return r
        retry_after = r.headers.get("retry-after", "")
        wait = float(retry_after) if retry_after.isdigit() else 2 ** attempt
        await asyncio.sleep(random.uniform(0, wait))   # full jitter
    raise RuntimeError(f"gave up on {url}")

client = httpx.AsyncClient(headers={"user-agent": UA}, follow_redirects=True,
                           timeout=httpx.Timeout(20, connect=5))

Identify the crawler in the User-Agent with a contact, cap concurrency per host, honour Retry-After, and back off on 429 and 503 rather than retrying at once. In Scrapy the same knobs are CONCURRENT_REQUESTS_PER_DOMAIN, DOWNLOAD_DELAY, AutoThrottle and ROBOTSTXT_OBEY (on in a generated project).

Re-crawling without re-downloading

“Re-crawl a million pages nightly” is mostly about not fetching or storing what did not change:

• Conditional GET: send If-None-Match with the stored ETag, or If-Modified-Since with Last-Modified; an unchanged page answers 304 Not Modified with no body.
• Sitemaps and feeds: lastmod in sitemap.xml tells you which pages changed, without fetching them.
• Hash the extracted content, not the HTML. Pages embed timestamps, rotating ads and CSRF tokens, so a hash of the raw response changes on every crawl. Hash the normalised fields you extract.
• Adaptive revisits: pages that rarely change are visited less often.

Idempotent storage

Canonicalise URLs first (lowercase host, drop the fragment and utm_* parameters, sort the query, honour rel=canonical), then upsert and write only when the content changed:

INSERT INTO pages (url, content_hash, data, fetched_at)
VALUES (:url, :hash, :data, now())
ON CONFLICT (url) DO UPDATE
   SET content_hash = EXCLUDED.content_hash,
       data = EXCLUDED.data, fetched_at = EXCLUDED.fetched_at
 WHERE pages.content_hash IS DISTINCT FROM EXCLUDED.content_hash;

Keep the raw responses (WARC files, or compressed blobs in object storage) so that fixing a parser bug means re-parsing, not re-fetching a million pages. Feeding a RAG index has its own delete and orphan rules: Ingestion pipelines and keeping an index in sync.

Playwright as the integration tool

When a page truly needs rendering, Playwright is the current default: auto-waiting removes most timing flakiness, and one API drives Chromium, Firefox and WebKit.

async def skip_heavy(route):                      # images, fonts, media: not needed
    if route.request.resource_type in ("image", "font", "media"):
        await route.abort()
    else:
        await route.continue_()

async with async_playwright() as p:
    browser = await p.chromium.launch()
    ctx = await browser.new_context(storage_state="auth.json")  # reuse login
    page = await ctx.new_page()
    await page.route("**/*", skip_heavy)
    async with page.expect_response("**/api/orders*") as resp:
        await page.goto(URL)
    orders = await (await resp.value).json()     # the page's own JSON, not the DOM

• Reuse authenticated state (storage_state): logging in on every run is slow and is what triggers rate limits and challenges.
• Capture the page’s own API response with expect_response rather than parsing the rendered DOM; when you must use the DOM, select by role and text (get_by_role("button", name="Export")), not CSS paths that break on any redesign.
• One browser, many contexts: a context is cheap; a browser per job is hundreds of MB. Do not use networkidle as a readiness signal on pages with polling or analytics; wait for the element or response you need.
• Reap Chromium in containers (docker run --init, or tini as PID 1), or crashed renderers accumulate as zombies.
• Capture evidence on failure: a screenshot and a Playwright trace (DOM snapshots and network per step) turn “it stopped working” into an answer.

Anti-bot defences, and what they see

Commercial bot management (Cloudflare Bot Management and Turnstile, Akamai, DataDome, HUMAN) scores every request on layers that a headless scraper fails one by one:

Two consequences for the design:

• Blocks are often soft: a 200 with a challenge page, an empty result list, or a decoy page (Cloudflare’s AI Labyrinth serves generated pages to suspected bots). Success must be asserted on the content, never on the status code.
• AI crawlers are blocked by default on Cloudflare sites since 2025-07-01, with an opt-in “pay per crawl” that answers HTTP 402 with a price. For a crawler feeding a model, this is increasingly a commercial negotiation, not a technical problem.

Proxies

Rotate per session, not per request, when the site tracks sessions: an IP changing mid-flow looks like exactly the abuse detection exists to catch. Run the pool like any dependency: health checks, retire proxies that start returning blocks, cap concurrency per domain. And if the URLs to fetch come from users, the fetcher is an SSRF vector into your network (SSRF and Command Injection).

The legal picture

Not an engineer’s call alone, but a senior engineer knows the landmarks:

• US, CFAA: Van Buren (2021) and hiQ v. LinkedIn narrowed “unauthorised access” for public pages, but hiQ still lost in 2022 on breach of LinkedIn’s user agreement. Meta v. Bright Data (2024): scraping public data while logged out did not breach Meta’s terms. Logged-in scraping is bound by the terms you accepted.
• EU, databases and contracts: Ryanair v PR Aviation (CJEU, 2015): terms of use can bind access to a database that has no copyright protection.
• Personal data: GDPR applies to scraped personal data regardless of it being public; Clearview AI was fined by several EU regulators.
• Text and data mining: the EU DSM Directive (Article 4) allows TDM unless the rightsholder opts out in machine-readable form, and the AI Act requires general-purpose AI providers to respect those opt-outs.

Proxies are appropriate for data you are entitled to, at a volume one address cannot sustain; they are not a route around an explicit prohibition. For a partner platform, asking for API access first often removes the whole problem.

Run it like a queue consumer

Never inside a request handler: fetching and especially rendering are slow, memory-hungry and fail often.

• A job per unit of work, idempotent, so a retry is harmless.
• Concurrency capped per host and, for browsers, by memory.
• A dead-letter queue for jobs that keep failing, watched by a human.
• Crawl traps (calendars, faceted navigation, session ids in URLs) bounded by depth limits and a per-host URL cap.

See arq and dramatiq: the Celery alternatives.

Where it breaks in production

• Selector drift: a redesign makes a field None on every page. Validate each item with Pydantic at the boundary and alert on the null rate per field.
• Soft bans: 200s with a challenge or empty list; track items per page and the ban rate per proxy and per domain.
• Silent shrinkage: the crawl “succeeds” with fewer items each night. Alert on success rate and item counts, not on errors.
• Charset or redirect bugs: mojibake from a missing charset; 301 bodies stored as content.
• A robots.txt misread: first-match parsing on Python before 3.14 crawls what the site disallowed.
• Memory growth: unclosed browser contexts and zombie renderers.

Dashboard per host: status-code mix, items per page, freshness lag.

Related

• Designing an external API integration - designing the integration layer
• aiohttp — Timeouts, Retries, and Pooling Patterns - client timeouts, retries, pools
• Proxies: Forward vs Reverse - how forward proxies work
• E2E Testing with Playwright - Playwright for its original purpose
• Idempotency keys - why the job must be replayable