How to scrape product prices with Python without getting blocked

Scraping a product price sounds like a ten-minute job. Fetch the page, grab the element, print the number. And the first version usually does work — on your laptop, on one store, once. The trouble starts on day three, when the same script returns an empty string, a CAPTCHA, or a 403.

The gap between a scraper that runs and a scraper that keeps running is where most of the work hides. Here's what that actually involves.

Why your first scraper stops working

Almost every failure comes down to one of four things:

• The price is rendered by JavaScript, so it simply isn't in the HTML you downloaded.
• The site fingerprints your request — no real browser sends the headers a bare HTTP client sends.
• You're hitting from one IP address, fast, forever. That pattern is trivial to spot.
• The markup changed. A class name moved and your selector now points at nothing.

Each one has a known fix. The problem is that you need all four, and you need them at the same time.

Start by checking where the price actually lives

Before writing any parsing code, open the page, disable JavaScript, and reload. If the price disappears, you're not scraping HTML — you're scraping an API. Open the network tab and look for the JSON request that carries the price.

This single check saves hours. Hitting the underlying JSON endpoint is faster, far more stable, and doesn't break every time the design changes. A rendered page is the fallback, not the default.

Look like a real browser

A default Python request announces itself immediately. At minimum, send a realistic User-Agent, an Accept-Language, and reuse a session so cookies persist between requests:

import requests

session = requests.Session()
session.headers.update({
    "User-Agent": (
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
        "AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0 Safari/537.36"
    ),
    "Accept-Language": "en-US,en;q=0.9",
})

response = session.get(url, timeout=20)

This gets you past the laziest checks. It will not get you past a real anti-bot service — nothing simple will.

Slow down, and vary

Rate limiting is the part people skip and then regret. Requests every 0.2 seconds, forever, from one address, is not a traffic pattern any human produces. Add a randomised delay, cap your concurrency, and back off when you see a 429 instead of hammering harder.

If you need volume, you need rotating proxies — which means a proxy provider, a rotation strategy, and handling the fact that a chunk of any proxy pool is dead at any moment.

Write selectors that survive a redesign

Never anchor to generated class names. Prefer, in order:

1. 1Structured data — many stores ship schema.org Product/Offer JSON-LD with the exact price. Parse that first.
2. 2Stable attributes like data-testid or itemprop.
3. 3Text patterns as a last resort, with a currency-aware regex.

And make it fail loudly. A scraper that silently writes empty rows is worse than one that crashes — you'll trust the data for weeks before noticing.

The part nobody budgets for: maintenance

Writing the scraper is maybe 20% of the work. The other 80% is the months afterwards: the site redesigns, your proxies expire, a new bot check appears, your cron job dies quietly. That's the real cost, and it's why so many internal scrapers end up abandoned.

Be honest about whether to build it

Write it yourself when the target is simple and stable, the logic is specific to your business, or you actually want to own it. Don't write it when it's a solved problem you're re-solving — a price tracker for a common platform has been built thousands of times, and yours won't be meaningfully different.

AI will happily generate a scraper for you in seconds. What it won't do is know which of the four failure modes above will hit you, or be there in six weeks when the selector breaks. That's the gap a ready-made, maintained script fills.