Filter URLs with AI-powered robots.txt compliance & source verification
Ready for 3.0Ready for 3.0. This template uses no nodes that n8n 3.0 removes. No removed nodes, versions or helpers found. Static check, last run 2026-10-19.
What this workflow does
This AI workflow has 37 nodes and runs on a schedule (every day) (plus 1 other trigger). Along the way it connects Model Selector, Mistral Cloud Chat Model, Groq Chat Model and Google Gemini Chat Model and 1 more.
How it runs
Trigger: On a schedule (every day) (plus 1 other trigger).
- Start (Tests) (Edit Fields (Set)): sets or maps fields
- Get Base URL (Code): runs custom JavaScript code
- Create forbidden urls table (Postgres): uses Postgres
- Check forbidden urls table (Postgres): uses Postgres
- If: checks a condition and branches into true / false
- Return Disallow (Edit Fields (Set)): sets or maps fields
- Create robots.txt Table (Postgres): uses Postgres
- Check robots.txt Table (Postgres): uses Postgres
- If: checks a condition and branches into true / false
- Prepare Robots.txt Check (Edit Fields (Set)): sets or maps fields
…plus 21 more steps. See the full node list below.
Services it uses
Before you upgrade to n8n 3.0
Nothing in this workflow is removed in n8n 3.0, so it should import and run after the upgrade.
Nodes used (12)
- Schedule Trigger
n8n-nodes-base.scheduleTrigger - Code
n8n-nodes-base.code - Edit Fields (Set)
n8n-nodes-base.set - Information Extractor
@n8n/n8n-nodes-langchain.informationExtractor - Model Selector
@n8n/n8n-nodes-langchain.modelSelector - Mistral Cloud Chat Model
@n8n/n8n-nodes-langchain.lmChatMistralCloud - Groq Chat Model
@n8n/n8n-nodes-langchain.lmChatGroq - Google Gemini Chat Model
@n8n/n8n-nodes-langchain.lmChatGoogleGemini - HTTP Request
n8n-nodes-base.httpRequest - If
n8n-nodes-base.if - Postgres
n8n-nodes-base.postgres - Execute Workflow Trigger
n8n-nodes-base.executeWorkflowTrigger
From the author
URL Officer - Respect robots.txt and Avoid Undesirable Sources 🎬 Overview Version : 1.0 The URL Officer workflow automates the filtering of URLs by checking them against a database of forbidden sources and the rules defined in robots.txt files. It…