Evaluation metric: summarization
Ready for 3.0Ready for 3.0. This template uses no nodes that n8n 3.0 removes. No removed nodes, versions or helpers found. Static check, last run 2026-10-19.
Nodes used (11)
- Evaluation
n8n-nodes-base.evaluation - OpenAI Chat Model
@n8n/n8n-nodes-langchain.lmChatOpenAi - Evaluation Trigger
n8n-nodes-base.evaluationTrigger - No Operation, do nothing
n8n-nodes-base.noOp - Google Gemini Chat Model
@n8n/n8n-nodes-langchain.lmChatGoogleGemini - Structured Output Parser
@n8n/n8n-nodes-langchain.outputParserStructured - Google Drive
n8n-nodes-base.googleDrive - Basic LLM Chain
@n8n/n8n-nodes-langchain.chainLlm - Extract from File
n8n-nodes-base.extractFromFile - Webhook
n8n-nodes-base.webhook - Edit Fields (Set)
n8n-nodes-base.set
What it does
This n8n template demonstrates how to calculate the evaluation metric "Summarization" which in this scenario, measures the LLM's accuracy and faithfulness in producing summaries which are based on an incoming Youtube transcript. The scoring approach is adapted from https://cloud.google.com/vertex-ai/generative-ai/docs/models/metrics-templatespointwisesummarizationquality How it works This evaluation works best for an AI summarization workflows. For our scoring, we simple compare the generated response to the original transcript. A key factor is to look out information in the response which is not mentioned in the documents. A high score indicates LLM adherence and alignment whereas a low score could signal inadequate prompt or model hallucination. Requirements n8n version 1.94+ Check out this Google Sheet for a sample data https://docs.google.com/spreadsheets/d/1YOnu2JJjlxd787AuYcg-wKbkjyjyZFgASYVV0jsij5Y/edit?usp=sharing