Scrape and store data from multiple website pages

Needs changes

by Miquel Colomer ✓ · 107,228 views · created 2021-05-07 · Engineering

Needs changes. This template uses 5 thing(s) that n8n 3.0 removes: HTML Extract, Function (legacy), Function Item (legacy), Read Binary File, Write Binary File. 5 breaking item(s) found. Static check, last run 2026-10-19.

What this workflow does

This workflow has 23 nodes and runs manually, when you click “Execute workflow”. Along the way it connects MongoDB and uProc. It also calls theswiftcodes.com over HTTP.

How it runs

Trigger: Manually, when you click “Execute workflow”.

Create Directory (Execute Command): runs a shell command on the n8n host
HTTP Request: calls theswiftcodes.com
HTML Extract: uses HTML Extract
Prepare countries (Function): runs custom JavaScript (legacy Function node)
SplitInBatches (Loop Over Items (Split in Batches)): loops over the items in batches
uProc: uses uProc
Set Page to Scrape (Function Item): runs custom JavaScript per item (legacy Function Item node)
Set (Edit Fields (Set)): sets or maps fields
Generate filename (Function Item): runs custom JavaScript per item (legacy Function Item node)
Read Binary File: uses Read Binary File

…plus 12 more steps. See the full node list below.

Services it uses

MongoDB uProc

HTTP calls to: theswiftcodes.com

Scraping pages by hand? Firecrawl is a simpler option (10% off for new users).

Before you upgrade to n8n 3.0

5 things in this workflow are removed in n8n 3.0. Fix them before you upgrade:

HTML Extract in HTML Extract, HTML Extract1: replace with HTML node, 'Extract HTML Content' operation.
Function (legacy) in Prepare Documents, Set More Pages, Wait…: replace with Code node, mode 'Run Once for All Items'.
Function Item (legacy) in Set Page to Scrape, Generate filename: replace with Code node, mode 'Run Once for Each Item'.
Read Binary File in Read Binary File, Read Binary File1: replace with Read/Write Files from Disk.
Write Binary File: replace with Read/Write Files from Disk.

Rather not rewrite it? Many of these flows can be rebuilt in Make (free plan).

Removed in 3.0	Replace with
HTML Extract HTML Extract, HTML Extract1	HTML node, 'Extract HTML Content' operation
Function (legacy) Prepare Documents, Set More Pages, Wait, Prepare countries	Code node, mode 'Run Once for All Items'
Function Item (legacy) Set Page to Scrape, Generate filename	Code node, mode 'Run Once for Each Item'
Read Binary File Read Binary File, Read Binary File1	Read/Write Files from Disk
Write Binary File Write Binary File	Read/Write Files from Disk

Nodes used (13)

Manual Triggern8n-nodes-base.manualTrigger
HTTP Requestn8n-nodes-base.httpRequest
HTML Extractn8n-nodes-base.htmlExtract
Loop Over Items (Split in Batches)n8n-nodes-base.splitInBatches
MongoDBn8n-nodes-base.mongoDb
uProcn8n-nodes-base.uproc
Functionn8n-nodes-base.function
Ifn8n-nodes-base.if
Function Itemn8n-nodes-base.functionItem
Edit Fields (Set)n8n-nodes-base.set
Read Binary Filen8n-nodes-base.readBinaryFile
Write Binary Filen8n-nodes-base.writeBinaryFile
Execute Commandn8n-nodes-base.executeCommand

From the author

This workflow allows extracting data from multiple pages website. The workflow: 1) Starts in a country list at https://www.theswiftcodes.com/browse-by-country/. 2) Loads every country page 3) Paginates every page in the country page. 4) Extracts data from the…
Excerpt from Miquel Colomer's description on n8n.io.

View the original on n8n.io → Check your own copy

Scrape and store data from multiple website pages

What this workflow does

How it runs

Services it uses

Before you upgrade to n8n 3.0

Nodes used (13)

From the author

Get the weekly “what broke” email