Scraping a few dozen pages from a website is simple. You write a short script with Python and BeautifulSoup, loop through the URLs, and save the data to a file. Even if your script is inefficient, it will probably finish in a few minutes, and nobody will notice.
Everything changes when the scale jumps to 90,000 targets every day.
That was the challenge on WalletInvestor, a financial forecasting site covering crypto, stocks, forex, and commodities. The goal was to collect current market prices, historical series data, forecast targets, and chart images for roughly ninety thousand symbols on a recurring schedule.
When you do the math on a workload of that size, naive solutions break immediately. If you spin up a headless browser like Chromium via Playwright for all 90,000 pages, the execution time is completely unreasonable, and the proxy bandwidth bill will drain your budget before the first week is over. Every browser instance downloads megabytes of scripts, styles, and tracking pixels that you do not need.
On the other hand, relying entirely on simple HTTP requests and HTML parsing does not work either. The tabular price numbers are in the static HTML, but the interactive forecast graphs are rendered dynamically through client-side JavaScript. A plain GET request just returns an empty container where the chart belongs.
To solve both problems without ballooning the proxy costs, I split the extraction engine into two distinct modes: Lite and Graph.
The Lite pipeline handled the bulk of the data. Written in Python with BeautifulSoup, it sent lightweight HTTP requests directly to the target pages. It extracted tabular prices, historical price series, and forecast milestones straight out of the HTML payload. Because it did not have to boot a browser runtime, it ran rapidly and consumed minimal bandwidth. We paired this mode with rotating proxy pools, custom user-agent headers, and adaptive request pacing to keep requests distributed evenly without triggering rate limits.
The Graph pipeline was reserved exclusively for targets where visual chart snapshots were mandatory. When a symbol required its forecast chart, the engine routed the request through Playwright. The browser loaded the page context, allowed the client-side JavaScript to render the interactive chart, and captured the snapshot before closing the session.
By separating the two concerns, we avoided loading heavy browser runtimes for data that was already sitting in plain HTML. The fast pages stayed fast, and the proxy bandwidth remained under control.
A pipeline this large cannot just be a local script you run in a terminal whenever you remember. It needs to behave like an operational service.
I wrapped the scraping core in a lightweight web server that exposed REST API endpoints. This allowed other systems and developers on the team to trigger jobs, check current pipeline status, and query extracted records directly from PostgreSQL. Crucially, operational settings like target queues, pacing intervals, and retry thresholds were configurable through the API at runtime. If target servers started responding slowly, we could dial back concurrency on the fly without stopping the process or editing configuration files.
Scheduling was configured via cron, timed specifically to Chicago market close (US Central Time) so data refreshes aligned with daily trading activity.
Visibility was the final piece. When you run long background batch jobs, you cannot sit and stare at logs. I built an automated reporting system that emailed run summaries at the start and conclusion of each job. The summaries reported total symbols processed, success percentages, and run durations. Most importantly, the service tracked cumulative proxy data usage and sent warning alerts well before bandwidth limits approached their thresholds.
Data engineering at scale is rarely about using the most complicated tool available. It is about understanding what each page actually needs, using the lightest tool that can get the job done, and building enough telemetry so you always know when things go wrong before your users do.