Backfill historical API data into Postgres with resumable checkpointed batches
Ready for 3.0What this workflow does
This workflow has 17 nodes and runs when an HTTP GET request hits its webhook URL (plus 1 other trigger). Along the way it connects Postgres.
How it runs
Trigger: When an HTTP GET request hits its webhook URL (plus 1 other trigger).
- Postgres: run a query
- Code: runs custom JavaScript code
- If: checks a condition and branches into true / false
- Code: runs custom JavaScript code
- HTTP Request: calls an HTTP API
- Wait: pauses before continuing
- Code: runs custom JavaScript code
- Send Data to Webhook (Respond to Webhook): sends the response back to the webhook caller
- If: checks a condition and branches into true / false
- Postgres: run a query
…plus 4 more steps. See the full node list below.
Services it uses
Before you upgrade to n8n 3.0
Nothing in this workflow is removed in n8n 3.0, so it should import and run after the upgrade.
Nodes used (9)
- Manual Trigger
n8n-nodes-base.manualTrigger - Postgres
n8n-nodes-base.postgres - Code
n8n-nodes-base.code - If
n8n-nodes-base.if - HTTP Request
n8n-nodes-base.httpRequest - Wait
n8n-nodes-base.wait - No Operation, do nothing
n8n-nodes-base.noOp - Webhook
n8n-nodes-base.webhook - Respond to Webhook
n8n-nodes-base.respondToWebhook
From the author
Quick overview This workflow runs on demand (manual or webhook) to backfill historical data from an HTTP API into Postgres in resumable batches, persisting a checkpoint and progress stats so repeated runs can continue from the last saved position. How it…