🌐 Firecrawl website content extractor

🌐 Firecrawl Website Content Extractor (n8n Workflow) This n8n automation workflow uses Firecrawl API to extract structured data (e.g., quotes and authors) from web pages — such as Quotes to Scrape — and handles retries in case of delayed extraction. --- 🔁 Workflow Overview 🎯 Purpose: - Crawl and extract structured web data using Firecrawl - Wait for asynchronous scraping to complete - Retrieve and validate results - Support retries if content is not ready --- 🔧 Step-by-Step Node Breakdown 1. 🧪 Manual Trigger - Node: When clicking ‘Test workflow’ - Used to manually test or execute the workflow during setup or debugging. --- 2. 📤 Firecrawl Extract API Request - Node: Extract - Sends a POST request to https://api.firecrawl.dev/v1/extract - Payload includes: - urls: List of pages to crawl - prompt: "Extract all quotes and their corresponding authors from the website." - schema: JSON schema defining expected structure (quotes, each with text and author) > 📌 Uses an HTTP Header Auth credential for Firecrawl API --- 3. ⏱️ Wait for 30 Seconds - Node: 30 Secs - Gives Firecrawl time to finish processing in the background - Prevents hitting the API before results are ready --- 4. 📥 Get Results - Node: Get Results - Performs a GET request to the status URL using {{ $('Extract').item.json.id }} to retrieve extraction results. --- 5. ✅❌ Condition Check - Node: If - Checks if the data array is empty (i.e., no results yet) - If data is empty: - Waits 10 more seconds and retries - If data is available: - Passes data to the next step (e.g., processing or storage) --- 6. 🔁 Retry Delay - Node: 10 Seconds - Waits briefly before sending another GET request to Firecrawl --- 7. 🛠️ Edit Fields (Optional Output Formatting) - Node: Edit Fields - Placeholder to structure or format the extracted results (quotes and authors) --- 🧾 Sticky Note: Firecrawl Setup Guide Included as an embedded reference: - 🔗 10% Firecrawl Discount - 🧰 Instructions to: - Add Firecrawl API credentials in n8n - Use Firecrawl Community Node for self-hosted instances - Set up the schema and prompt for targeted data extraction --- ✅ Key Features - 🔌 API-based crawling with schema-structured output - ⏱️ Smart waiting + retry mechanism - 🧠 AI prompt integration for intelligent data parsing - ⚙️ Flexible for different URLs, prompts, and schemas --- 📦 Sample Output Schema json { "quotes": { "text": "The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.", "author": "Albert Einstein" }, { "text": "It is our choices, Harry, that show what we truly are, far more than our abilities.", "author": "J.K. Rowling" } }