One search across many sources: how an aggregator works

An aggregator does four things: it collects listings from several sources, brings them into the same shape, removes the duplicates and gives you one search across all of them, with alerts when something new shows up. The visible part is the search box. The real work happens behind it, in steps two and three.

As an example, picture a rental aggregator: listings collected from the major portals, with a map, alerts and prices in euros.

1. Collecting, politely

The program that collects listings reads the sources' public pages at regular intervals. The first rule is to respect what each site asks for. The robots.txt file, described in the RFC 9309 standard from 2022, tells automated programs which parts of a site they may read. The standard says plainly that these rules are not a form of access authorization, so they don't replace an agreement, but they're the first thing you check. It also recommends not relying on a cached copy for more than 24 hours, so the rules get re-read often.

The second rule is legal. The EU Database Directive gives whoever invested in a database the right to prevent copying of all of it or a substantial part. It also bars repeated and systematic extraction of small parts when that undermines the database's normal use. That's why a fair aggregator shows a summary and sends people to the original listing instead of republishing everything. What gets collected from where is something to go over with a lawyer on every project.

2. The same shape for every listing

Every source writes things differently. One says "2 rooms," another "two-room," another "apartment with 2 bedrooms." Prices come in lei or euros, floor area with or without the balcony, neighborhoods under their official or everyday names. This step turns everything into one format: number of rooms, floor area, price in a common currency, map coordinates.

It sounds dull, but everything else depends on it. A filter like "2 rooms, under a certain amount, in area X" only works if every listing says the same things the same way.

3. Duplicates

The same apartment often shows up on three portals, posted once by the owner and twice by agencies. Show them all and the search fills with noise. To spot them, an aggregator compares several signals: floor area, floor, price, map position, photos. No single one is enough: two identical apartments in the same building have the same numbers but different photos.

It's better to group duplicates than to delete them. People see one listing with its sources underneath and choose where to click.

4. Search, map and alerts

Only now do you get the part users see: filters, a map, sorting. Clean data from the earlier steps makes search fast. It also enables things no single source has: price statistics by city and alerts for new listings matching a saved search. For someone hunting for a rental, the alert is often more useful than the search itself.

What Google expects from an aggregator

Google's spam policies list republishing content from other sites without adding original content or value as an abusive practice. An aggregator that only copies listings runs exactly that risk. The value comes from what you add: duplicates removed, the map, the statistics, the alerts, city pages with your own information.

There's a technical problem too. Filters create many URLs for nearly the same page. Google recommends choosing a preferred (canonical) URL for duplicate or very similar pages, through redirects or the rel="canonical" tag, so signals aren't spread across dozens of variants.

If you have an idea for an app that pulls data from several places, our App Dev services page shows how we work.