Queue OpenAI-compatible LLM chat requests with PostgreSQL and a local GPU

Quick overview An n8n-first, durable FIFO gateway for a local OpenAI-compatible language model. It accepts ordinary Chat Completions requests, queues them in PostgreSQL, runs one inference at a time, and supports human retrieval commands, automatic waiting by shell-capable AI harnesses, and OpenAI-compatible SSE response framing. How it works 1. Receives OpenAI-style requests on a POST /v1/chat/completions webhook secured with header authentication. 2. Validates and classifies the request as a new submission or a control command (get result/wait, cancel, or list queue), then stores new jobs in a PostgreSQL queue table. 3. Looks up, cancels, or lists queued/running jobs in PostgreSQL and returns an OpenAI-compatible JSON response or an OpenAI SSE-framed response when stream: true is requested. 4. Exposes a GET /v1/queue/tasks/:taskId webhook to return the durable status, queue position, and any stored error details for a specific task. 5. Exposes a GET /v1/models webhook to return the configured comma-separated model identifiers without calling the upstream LLM. 6. Runs every 5 seconds to recover expired leases, claim the oldest eligible queued job under PostgreSQL locks, call the upstream chat completions URL, and persist the success result or schedule a bounded retry back into PostgreSQL. Setup 1. Add a PostgreSQL credential, point all PostgreSQL nodes to it, and run the manual setup trigger once to create the llmqueuejobs table and indexes. 2. Configure header authentication on all three webhooks (/v1/chat/completions, /v1/queue/tasks/:taskId, and /v1/models) and deploy behind TLS. 3. Update the dispatcher configuration values (upstream chat completions URL, timeout, lease minutes, and retry base seconds) and set any required upstream authentication in the HTTP Request node. 4. Update the gateway configuration values (max attempts and the wait/poll limits) and set the model IDs returned by /v1/models. 5. Copy the webhook URLs and configure your OpenAI-compatible client or gateway to use them as its base URL and auth header. Additional info API Chat Completions POST /v1/chat/completions The body is the ordinary OpenAI Chat Completions request. Both stream: false and stream: true are accepted. The original body is stored in PostgreSQL; the dispatcher forces the upstream request to stream: false and removes streamoptions so the result can be persisted reliably. bash curl -sS "$OPENAIBASEURL/chat/completions" \ -H "Authorization: Bearer $OPENAIAPIKEY" \ -H 'Content-Type: application/json' \ -d '{ "model": "queued-local-model", "messages": {"role": "user", "content": "Explain transactional queues."}, "stream": false, "maxtokens": 800 }' | jq For SSE-compatible mode: bash curl -sS -N "$OPENAIBASEURL/chat/completions" \ -H "Authorization: Bearer $OPENAIAPIKEY" \ -H 'Content-Type: application/json' \ -d '{ "model": "queued-local-model", "messages": {"role": "user", "content": "Explain transactional queues."}, "stream": true, "streamoptions": {"includeusage": true}, "maxtokens": 800 }' Streaming semantics When stream: true, successful Chat Completions responses use Content-Type: text/event-stream. The gateway converts its queue acknowledgement, synthetic waiting tool call, or stored completion into OpenAI chat.completion.chunk events and terminates the response with data: DONE. This is SSE protocol compatibility, not live GPU-token streaming. The initial request still returns promptly after enqueueing. After a shell-capable harness reports that waiting has completed, the stored answer is replayed as SSE chunks. If streamoptions.includeusage is true, a final usage chunk with an empty choices array is emitted before DONE. Task status GET /v1/queue/tasks/:taskId bash curl -sS "$OPENAIBASEURL/queue/tasks/taskREPLACEME" \ -H "Authorization: Bearer $OPENAIAPIKEY" | jq Model discovery GET /v1/models Human flow A normal prompt returns immediately: text Your request is queued as task.... Check it with /result task.... The following messages a