Run incremental ETL from REST APIs to PostgreSQL with Slack schema alerts
Ready for 3.0What this workflow does
This workflow has 24 nodes and runs manually, when you click “Execute workflow” (plus 1 other trigger). Along the way it connects Postgres.
How it runs
Trigger: Manually, when you click “Execute workflow” (plus 1 other trigger).
- Postgres: run a query
- Set ETL Config (Edit Fields (Set)): sets 11 fields: pipelineName, sourceUrl, recordsPath…
- Postgres: run a query
- Postgres: run a query
- Code: runs custom JavaScript code
- HTTP - Extract Source Page (HTTP Request): calls an HTTP API
- Code: runs custom JavaScript code
- Switch: routes items into 2 branches
- Code: runs custom JavaScript code
- HTTP - Slack Extract Failure Alert (HTTP Request): calls an HTTP API (POST)
…plus 12 more steps. See the full node list below.
Services it uses
Before you upgrade to n8n 3.0
Nothing in this workflow is removed in n8n 3.0, so it should import and run after the upgrade.
Nodes used (8)
- Manual Trigger
n8n-nodes-base.manualTrigger - Postgres
n8n-nodes-base.postgres - Schedule Trigger
n8n-nodes-base.scheduleTrigger - Edit Fields (Set)
n8n-nodes-base.set - Code
n8n-nodes-base.code - HTTP Request
n8n-nodes-base.httpRequest - Switch
n8n-nodes-base.switch - If
n8n-nodes-base.if
From the author
Quick overview This workflow runs every 15 minutes to incrementally pull updated order records from a REST API into PostgreSQL, automatically mapping and adapting the destination table when the source schema changes, and sending Slack alerts for extract…