Scraping

The user-facing version of this feature is documented at URL scraping. This page is the operator’s reference: how the pipeline works, what the admin can configure, and how to add a new provider.

The scraping admin lives at Admin > Scraping. Everything except the env-var first-boot seeds is database-backed and takes effect without a restart.

The Pipeline at a Glance

  add-item dialog                    /admin/scraping
        |                                  |
        | paste URL                        | tune knobs
        v                                  v
  +-------------------------------------------------+
  | GET /api/scrape/stream (SSE)                    |
  |   - auth + URL validate                         |
  |   - build provider chain                        |
  +-------------------------------------------------+
                       |
                       v
  +-------------------------------------------------+
  | orchestrator                                    |
  |   - tiers (fall through until price + photo)    |
  |   - one running merge across all tiers          |
  |   - cache lookup / dedup                        |
  |   - post-passes (clean-title)                   |
  +-------------------------------------------------+
                       ^
                       |
   +-------------------+---------------------+
   | fetch-provider      scrapeProviders[]   |
   | (always on, free)   (admin-configurable)|
   +-----------------------------------------+

The always-on fetch-provider runs first. Everything else is opt-in. For Amazon it fetches the page as a regular browser first (Amazon hides prices from link-preview crawlers) and reads the price and full-size product photos from Amazon’s own markup.

Tiers

Each configured provider has a tier (1-5). The orchestrator runs all tier-1 entries in parallel, folds their results into one running merge, and only advances to tier 2 if that merge is still below the quality threshold. Same for tier 2 to tier 3, etc.

The implicit fetch-provider is tier 0; it runs first, alone.

Why tiers: cost control. Put cheap providers in tier 1 and paid hosted services in tier 2 and up. Pages where an early tier gets a complete result (title, price, and a real product photo) never spend money on Browserbase, ScrapFly, or AI extraction.

Default tier on new entries:

Provider type	Default tier
browserless	1
flaresolverr	1
custom-http	1
giftwrapt-scraper	1
browserbase-fetch	2
scrapfly	2
browserbase-stagehand	3
ai	3

Adjust the tier from the Tier dropdown on each card in /admin/scraping.

Provider Types

All configured at Admin > Scraping. Secret fields are encrypted at rest in app_settings (AES-256-GCM, master key derived from BETTER_AUTH_SECRET).

Type	Returns	Secret	Best for
browserless	HTML	token	Self-hosted JS rendering
flaresolverr	HTML	-	Self-hosted; resolves interstitials before returning HTML
browserbase-fetch	HTML	apiKey	Hosted JS rendering, no LLM
browserbase-stagehand	structured	apiKey	Hosted structured extraction with LLM
scrapfly	HTML	apiKey	Hosted rendering service
giftwrapt-scraper	HTML	token	A self-hosted GiftWrapt Scraper gateway (headless rendering)
custom-http	HTML or JSON	customHeaders	Bring-your-own scraper service
ai	structured	-	LLM extraction from raw HTML, using your AI provider

HTML responses go through the built-in extractor; structured ones are used as returned. You can add multiple entries of the same type (e.g. two Browserless instances) and drag to reorder within a tier.

A custom-http entry is called as GET <endpoint>?url=<encoded URL> with your custom headers. In JSON mode it must answer within the per-provider timeout with this shape (unknown fields are ignored, every field is optional):

{
  "title": "string",
  "description": "string",
  "price": "string",       // human-readable, e.g. "29.99"
  "currency": "string",    // e.g. "USD"
  "imageUrls": ["string"], // absolute URLs preferred
  "siteName": "string",
  "finalUrl": "string"     // the requested URL is used otherwise
}

How Results Are Merged

Every successful result, from every tier, feeds one running merge. Falling through to a later tier never throws away what an earlier tier found: if the built-in fetch got the photos and a tier-2 provider got the price, the final result has both.

• The highest-scoring result is the “base.”
• Any field the base left empty (title, description, price, currency, siteName, ratings) is filled from the others, best score first.
• imageUrls are pooled from everyone, then de-duplicated (different sizes of the same photo count as one, keeping the largest), ranked (real product photos first, social-share cards next, thumbnails last), and capped at the best 12.
• Before a result joins, a same-product check compares it with the base. Different Amazon product IDs, or titles that share too few words, keep it out, so a later tier that hit a captcha page or a different product can’t graft its data onto a good result.

The merged result is persisted as its own row with scraperId: 'merged:a,b,c', marked as the run’s result. The Scrape History card in /admin/scraping shows it with a Result badge directly above that run’s individual attempts, and its detail lists which providers contributed and which were kept out (and why).

Scoring and the Quality Threshold

Each result is scored by what it found. Price and a real product photo matter most:

Signal	Points
Product title (not the site name or an error page)	+2
Price	+3
Real product photo	+3
Social-share card as the only photo	+1
Description of 30+ characters	+1
Page looks like a bot or login wall	-3
Title looks like an error page (“404”, “Access Denied”)	-3

scrapeQualityThreshold (default 8) is the score the merge has to reach to stop trying further tiers. 8 is exactly title + price + real photo, so a result missing either keeps going while paid tiers are configured. Lower it to spend less on paid providers and accept results without a price; raise it to also require a description.

Each attempt’s score breakdown is stored with it and shown in its detail in Scrape History. The weights live in src/lib/scrapers/score.ts.

Upgrading From the Old Scale

Before this scale, results scored at most 6 and the default threshold was 3. A threshold saved under the old scale stops too early on the new one. Set it to 8 when you upgrade.

Dead Links

When a product URL redirects to the store’s homepage (or a locale homepage like /en-us, or a “store closed” password page), the product is gone. The scrape stops there instead of trying paid tiers, the add-item form shows This link is no longer valid., and a bulk-import job for it fails immediately instead of retrying. A URL that was already a homepage scrapes normally.

Cache

A successful scrape is reused for the same URL for scrapeCacheTtlHours. The cache serves the best recent run’s merged result, as long as it scored at least 5 (for example a title and a real photo, or a title and a price). That floor is separate from the quality threshold on purpose: a page that never shows a price can’t reach 8, but is still worth reusing rather than re-running every paid tier.

Knobs

Setting	Default	What it does
scrapeProviderTimeoutMs	20000	Per-provider HTTP budget
scrapeOverallTimeoutMs	45000	Overall scrape budget incl. parallel racers
scrapeQualityThreshold	8	Score needed to stop trying further tiers
scrapeCacheTtlHours	24	URL-based dedup cache TTL (0 = disable)
scrapeAiCleanTitlesEnabled	false	Post-pass: AI normalises the winning title

AI extraction itself is a regular provider entry (type ai) in a tier of your choosing. The legacy scrapeAiProviderEnabled toggle is only read at boot, to create that entry when none exists.

The AI features here depend on a configured AI provider. The toggles disable themselves in the admin UI if no AI provider exists.

Item Scrape Queue

Items can be imported in bulk from an external source (CSV, JSON, etc.). The scraper doesn’t run inline for those - imported rows get queued in item_scrape_jobs and drained by the /api/cron/item-scrape-queue cron tick. Three knobs control the drain:

Setting	Default	What
scrapeQueueUsersPerInvocation	25	Distinct users handled per cron tick
scrapeQueueConcurrency	3	Max parallel jobs per user inside one tick
scrapeQueueMaxAttempts	3	Max attempts before a job is marked failed
importEnabled	true	Master kill switch for the import flow

Failed jobs are kept for diagnostics. The parent item retains whatever fields were already set at create time. A dead link fails on its first attempt.

SSRF Protection

Authenticated users supply the URL, so the scraper has to refuse to fetch addresses inside your own infrastructure. The always-on fetch-provider and the AI provider both go through safeFetch, which:

• Rejects non-http(s) schemes.
• DNS-resolves the hostname and rejects any address in a private, loopback, link-local, CGNAT, multicast, or reserved range.
• Walks redirects manually (redirect: 'manual') and re-runs the same check on every hop.
• Caps the redirect chain (default 5).
• Sets credentials: 'omit'.

Hosted providers (Browserbase, ScrapFly, Browserless, etc.) make outbound calls from outside your infra, so they don’t need this hostname check on the user-supplied URL - their upstreams apply their own rules.

Encryption at Rest

Secret fields on configured providers (token, apiKey, customHeaders containing auth) are AES-256-GCM-encrypted in app_settings JSONB. The master key is derived via scrypt from BETTER_AUTH_SECRET. New writes encrypt; reads accept either an envelope or legacy plaintext.

Don't Rotate BETTER_AUTH_SECRET

Rotating BETTER_AUTH_SECRET invalidates every encrypted value in app_settings, including these provider secrets. You’d have to re-enter them all.

Deployment Matrix

Scenario	Working chain	Notes
Vercel, no extras	fetch-provider only	Best-effort; manual entry fallback.
Vercel + Browserbase Fetch	fetch -> browserbase-fetch	Cheapest hosted upgrade.
Vercel + ScrapFly	fetch -> scrapfly	Hosted rendering via asp=true. Pay per credit.
Vercel + Browserbase + Stagehand	fetch -> browserbase-fetch -> stagehand	Stagehand only runs when the earlier tiers lack a price or photo.
Vercel + custom HTTP	fetch -> custom-http	Bring-your-own scraper, no vendor cost.
Self-host containers	Full chain incl. browserless + flaresolverr	Run them as sidecars.
AI on top of any of the above	adds an ai entry, usually the last tier	Off by default. Costs money.

Adding a New Provider Type

1. Add a discriminated variant to scrapeProviderEntrySchema in src/lib/settings.ts. Mark secret fields with appSecretField().
2. If a new secret field type, include it in encryptScrapeProviderSecrets.
3. Create src/lib/scrapers/providers/<id>.ts exporting a create<Id>Provider(entry) factory.
4. Pick a sensible default tier in makeDefaultEntry() (cheap and self-hosted low, paid high).
5. Wire the factory into loadConfiguredProviders().
6. Add a card component in scraper-providers-form-view.tsx plus a default entry shape in makeDefaultEntry().
7. Add a fixture test covering the success path and each error code.

Errors should classify into the small enum (bot_block, http_4xx, http_5xx, network_error, timeout, invalid_response, config_missing, dead_link, unknown) via ScrapeProviderError so the orchestrator surfaces the right wire code.