When scraping modern travel sites, the biggest mistake you can make is trying to parse the page DOM.
On platforms like Expedia, pages are not static HTML files. They are single-page applications that load content through internal GraphQL APIs as the user scrolls. The markup is packed with dynamically generated CSS class names, deeply nested wrapper divs, and anti-bot verification scripts. If you run a standard headless browser script, Cloudflare or similar bot detection systems will usually block you before the first review renders.
Even if you manage to load the page, writing selectors to scrape review text, reviewer tags, stay dates, and sub-ratings from the HTML is a maintenance headache. The moment the site updates its frontend layout, your selectors break.
I took a different approach when building an extraction pipeline for a large hotel queue on Expedia. Instead of fighting the frontend markup, I used Playwright to let the browser do what it does naturally, intercepted the network requests underneath, and threw away the visual clutter.
The first step was getting past the bot verification. Headless browsers have subtle fingerprint tells that modern security layers detect easily, including missing browser features, canvas quirks, and unnatural timing. Running Playwright in a headed browser environment, where an actual graphical window opens and executes, bypassed these checks consistently.
Running a headed browser across thousands of listings introduces a severe problem: bandwidth cost. If a real browser visits a hotel page, it downloads high-resolution photo galleries, third-party tracking tags, marketing pixels, and custom web fonts. When you route that traffic through residential proxies, your proxy budget evaporates in hours.
To solve this, I set up request interception rules in Playwright. Every outgoing network request passed through a filter:
If the request was for an image, stylesheet, font file, or known analytics script, Playwright aborted the request immediately. The browser never downloaded the heavy media. If the request was an internal API call carrying hotel reviews or stay metadata, Playwright let it through, waited for the response, and captured the raw JSON payload directly from the network stream.
This gave us the best of both worlds. The browser satisfied the anti-bot checks because it ran in a real headed window with authentic browser characteristics. At the same time, proxy bandwidth remained focused on the actual review data. Best of all, we never had to scrape the DOM. The data arrived as clean, structured JSON directly from the platform's own internal API responses, complete with exact review scores, travel categories, and room types.
The extraction engine was only half the project. The client needed a way to operate the system on their own without using terminal commands.
I built a web control dashboard that gave the client direct oversight of the pipeline. The interface showed live progress metrics, including collected reviews per hotel, overall run completion rates, and error logs. The client could start, pause, or resume jobs, adjust concurrency, and re-queue failed URLs with specific retry limits.
Because datasets at this scale can be awkward to work with in a single file, I added export logic that automatically split output files by target file size so the client could open them comfortably in spreadsheet tools. To protect long-running jobs from unexpected failures, the dashboard triggered scheduled database backups at fixed milestones, saving everything into PostgreSQL.
Deploying the system required one final practical decision. Headed browser automation relies on display server dependencies like X11 or Wayland, which can be finicky and unstable on headless Linux instances. Because the client already operated a Windows VPS, I deployed the complete system directly into their Windows environment. I configured the execution runtime, local PostgreSQL storage, and the web control panel so the client could log into their server through remote desktop and manage runs with simple clicks.
Scraping dynamic sites is rarely about finding a magic trick that fools bot detectors. Often, the cleanest solution is to let a real browser satisfy the platform's requirements, cut out the assets you do not need, and capture the data directly from the network.