Spanish real-estate market aggregator

Problem

Extraction ran on Apify, a third-party provider costing around €400 a month. The cost was the visible part; the real problem was underneath it. In a product whose value is the data, extraction is not a supporting service — it is the core. Their limits, their pricing and how fast they reacted when a portal changed were setting ours.

Decision

Replace it with our own extraction and processing pipeline, built as one adapter per portal rather than a single module full of conditionals.

The alternative was keeping the provider and writing only the normalisation layer. We ruled it out because it left the dependency untouched: we would still be waiting on someone else to fix a broken portal.

Implementation

Every portal implements a shared interface inside its own adapter. Adding a new one means writing it and registering it; the core — normalisation, deduplication, price history — does not change.

Four extraction techniques run side by side, one per what each portal actually exposes: JSON embedded in the HTML, JSON-LD structured data, reading Livewire’s serialised state, and parsing server-rendered HTML.

Portals behind bot protection go through a Playwright browser bridge. That is the expensive route — a real browser costs memory and time — so it is declared per adapter and paid only where there is no alternative, not across all seven.

The same property shows up on several portals with different descriptions and floor areas rounded differently. Everything is normalised into one schema and deduplicated so a flat is one row, not five. The price history is built on top of that.

Result

Seven portals in production over roughly 900 verified listings. Extraction cost down from €400 a month to between €20 and €35.

And the part that never shows on an invoice: when a portal changes, the fix is ours to make and it happens the same day. In exchange, maintenance became our own development time.

The product ships as a commercial API with Stripe payments.