Templates / Scrape every URL on the web without getting blocked by Anti-Bot technologies with Scrappey

Scrape every URL on the web without getting blocked by Anti-Bot technologies with Scrappey

Ready for 3.0

by Bela ✓ · 28,186 views · created 2024-06-25 · Market Research

Ready for 3.0. This template uses no nodes that n8n 3.0 removes. No removed nodes, versions or helpers found. Static check, last run 2026-10-19.

What this workflow does

This workflow has 3 nodes and runs on a schedule (every day). It uses only n8n's built-in nodes. It also calls publisher.scrappey.com over HTTP.

How it runs

Trigger: On a schedule (every day).

Test Data (Edit Fields (Set)): sets 2 fields: name, url
Scrape website with Scrappey (HTTP Request): calls publisher.scrappey.com (POST)

HTTP calls to: publisher.scrappey.com

Before you upgrade to n8n 3.0

Nothing in this workflow is removed in n8n 3.0, so it should import and run after the upgrade.

Nodes used (3)

Schedule Triggern8n-nodes-base.scheduleTrigger
Edit Fields (Set)n8n-nodes-base.set
HTTP Requestn8n-nodes-base.httpRequest

From the author

Purpose of the workflow Most scraping workflows get blocked by anti-bot technologies. To avoid this, you can use Scrappey to scrape every website you want. How it works: We use Test Data and make a API Call to the Scrappey service. We get the scraped website…
Excerpt from Bela's description on n8n.io.

View the original on n8n.io → Check your own copy