Scraper

Updated by Darryl Waterhouse on 27 October 2026 4 min read

The Scraper collects information from other websites into your own site - prices from a supplier's list, events from a listings page, articles from a news page. You show it one page, click the things you want, and it keeps them in a collection you can display, filter and export like any other. It can run again by itself every hour, day or week.

What it does

Point and click: no code. Click one of the things that repeat on a page (a product, a row, an article), then click its parts - its title, price, link, picture.
More than one page: follow the site's "next page" link, or go through page numbers in the address.
Runs in the background: start a run and leave the screen. You can see its progress and stop it.
Keeps your data tidy: running again updates the rows it already has instead of adding them twice.
Polite: it respects each site's robots.txt, waits between pages and reads one site at a time.

Getting started

Open Scraper from the admin sidebar and click New recipe in the bar at the top.
Type the address of the page and click Fetch page. The page appears with its scripts switched off.
Item: click one of the things that repeat - say, one product. Every product on the page is outlined. If only part of it is outlined, click Bigger box.
Fields: click the parts of a product you want. Each becomes a column. A heading becomes title; clicking a link's text also takes its address; a price becomes a number.
Optional: click Next page link, then the site's "next" link, so a run reads the following pages too.
Check the Preview: it shows exactly what a run will store.
Give the recipe a name and click Save and run.

The rows go into a collection named after the recipe (for example scrape-competitor-prices). Show them on a page with the [collection] shortcode, or open them from the recipe's menu with Show the rows.

Using it

Recipes shows each recipe as a card with its last run. Click a card to open it in the picker. Right-click it (or use its ⋮ button, or press Shift+F10) for Run now, Runs (each run's report: pages, rows, what changed and any pages that failed), Show the rows, Export and Delete.
While a run goes on, a line at the top shows its progress, with Stop. The sidebar badge pulses while one runs and turns red when a recipe's last run failed - hover it for which.
Fields: change a column's name, or what it takes - its text, a link's address, an image's address, an attribute (such as title or data-id) or a number. in 18 of 20 means two items do not have it. Right-click a field for Match rows on a re-run by this, Move up/down and Remove.
Item - Match more and Match fewer loosen or tighten which things count as items. Pick again starts the item over. You can also edit any rule by hand (they are CSS selectors).
Unsaved changes are kept in your browser: open the recipe again and you are asked whether to keep them.
Import / Export (in the bar at the top): save recipes to a file and load them on another site. Imported recipes start with their schedule off.
Deleting a recipe keeps the rows it collected; delete the collection in Collections if you want them gone too.

Recipe settings

Setting	What it does
Match rows on a re-run by	A column that is different for every row and stays the same between runs - a product's link is ideal. Re-runs update that row instead of adding it again. All values together treats a row that changed at all as new.
On a re-run	Update rows by key (the usual), Only add new rows, or Replace everything - the collection holds only what the latest run found, and only when that run read every page.
Run	By hand only, every hour, every day or every week.
At most this many pages a run	A run stops here whatever the paging says.

Scraper settings

The cog in the bar at the top of the recipes screen.

Setting	What it does
Wait between pages	Seconds between two requests to a site (two is a good default). A site's own Crawl-delay wins when longer.
Most pages any run may read	The ceiling for every recipe's page limit, up to 1000.
Respect each site's robots.txt	On by default. On a hosted site it is always on and the wait is at least one second.

Good to know

Some sites build their pages with JavaScript after they load. The Scraper reads pages as the site sends them, so on those sites it may find nothing - Scraper Pro reads pages as a browser shows them.
Icon fonts on some sites do not show in the picker (you may see empty squares). That is only the picker's display.
Only collect what you are allowed to: check a site's terms before scraping it.