Scrape, search and browse the web with a Firecrawl AI agent webhook

Turn any prompt into structured web data. Send a POST request with a natural language prompt and an optional JSON schema, and get back clean, structured results scraped from the web by an AI agent powered by Firecrawl. Use Cases - Data Enrichment: Feed company names or URLs from your CRM and get back structured firmographic data (industry, funding, team size, tech stack). - Lead Generation: Ask the agent to find pricing, contact pages, or product details for a list of competitors. - Market Research: Extract structured pricing plans, feature comparisons, or product catalogs from any website. - Content Aggregation: Pull structured news, events, or job postings from across the web on a schedule. - Sales Intelligence: Enrich prospect lists with company info, recent news, or tech stack details before outreach. How It Works POST /webhook/scrape-agent 1. Receive Scrape Request receives a POST request with prompt and an optional outputschema. 2. Validate Output Schema checks the schema. If none is provided, it falls back to a permissive default. If the schema is malformed, it returns a clear error via Return Schema Error. 3. Research & Extract Web Data takes the prompt and uses the full Firecrawl toolkit to research the web: - Search (/search): Finds relevant pages and sources across the web. - Scrape (/scrape): Extracts clean, structured content from any URL. - Interact (interactContext, interact, interactStop): Lets the agent interact with scraped pages in a live session. After scraping a page, the agent can click buttons, fill forms, navigate dynamic content, and extract data that static scraping cannot reach, all without managing sessions manually. This combination gives the AI agent complete web navigation capabilities. It can discover sources, read pages, and interact with dynamic content autonomously. 4. Format Response to Schema (Structured Output Parser) formats the agent's response to match the provided (or default) schema. 5. Return Structured Results sends the structured JSON back to the caller. Setup Requirements - Firecrawl API Key: Sign up at firecrawl.dev and grab your API key. Connect it in the Firecrawl credential nodes. - LLM Provider: Configure your Primary Chat Model and Fallback Chat Model nodes (e.g., OpenRouter, OpenAI, Anthropic). The template uses two model nodes for reliability, plus a separate Parser Chat Model for the output parser. - n8n Instance: Self-hosted or cloud. Make sure the webhook node is set to accept POST requests. API Reference Endpoint POST https://your-n8n-instance/webhook/scrape-agent Request Body | Field | Type | Required | Description | |-------|------|----------|-------------| | prompt | string | Yes | Natural language instruction for the agent | | outputschema | object | No | JSON Schema defining the desired output structure | Response Returns a JSON object matching the provided schema, or a flexible object if no schema was given. --- Testing Examples 1. Basic Request (No Schema) The agent decides the output structure on its own. bash curl -X POST "https://your-n8n-instance/webhook/scrape-agent" \ -H "Content-Type: application/json" \ -d '{ "prompt": "Find the latest pricing for Firecrawl" }' | jq Expected output: A JSON object with whatever structure the agent finds most appropriate for the data. Since no schema was provided, the internal default ({ "type": "object", "additionalProperties": true }) is used. 2. Request With a Custom Schema You define exactly the shape of data you want back. bash curl -X POST "https://your-n8n-instance/webhook/scrape-agent" \ -H "Content-Type: application/json" \ -d '{ "prompt": "Find the latest pricing for Firecrawl", "outputschema": { "type": "object", "properties": { "source": { "type": "string" }, "plans": { "type": "array", "items": { "type": "object", "properties": { "name": { "type": "string" }, "price": { "type": "string" }, "credits": { "type": "string" }, "highlights": { "type": "array", "items": { "type": "string" } } } } } } } }' | jq Expect