Evaluations metric: answer similarity

by Jimleuk ✓ · 1,299 views · created 2025-05-27 · Engineering, AI Summarization

What this workflow does

This AI workflow has 18 nodes and runs on Evaluation events (plus 1 other trigger). Along the way it connects OpenAI Chat Model and Evaluation. It also calls api.openai.com over HTTP.

How it runs

Trigger: On Evaluation events (plus 1 other trigger).

1. Remap Input (Edit Fields (Set)): sets 1 field: chatInput
2. AI Agent: runs an AI agent, with model: OpenAI Chat Model
3. Evaluation: check if evaluating
4. Set Input Fields (Edit Fields (Set)): sets 3 fields: question, answer, groundTruth
5. Get Embeddings (HTTP Request): calls api.openai.com (POST)
6. GroundTruth to Items (Split Out): splits a list into separate items
7. Remap Embeddings (Edit Fields (Set)): sets 1 field: answer
8. Get Embeddings (HTTP Request): calls api.openai.com (POST)
9. Create Embeddings Result (Merge): merges the data from its input branches
10. Remap Embeddings (Edit Fields (Set)): sets 1 field: data

…plus 4 more steps. See the full node list below.

Services it uses

OpenAI Chat ModelEvaluation

HTTP calls to: api.openai.com

Before you upgrade to n8n 3.0

Nodes used (12)

• OpenAI Chat Model@n8n/n8n-nodes-langchain.lmChatOpenAi
• Evaluation Triggern8n-nodes-base.evaluationTrigger
• Edit Fields (Set)n8n-nodes-base.set
• Evaluationn8n-nodes-base.evaluation
• No Operation, do nothingn8n-nodes-base.noOp
• AI Agent@n8n/n8n-nodes-langchain.agent
• Chat Trigger@n8n/n8n-nodes-langchain.chatTrigger
• HTTP Requestn8n-nodes-base.httpRequest
• Split Outn8n-nodes-base.splitOut
• Aggregaten8n-nodes-base.aggregate
• Mergen8n-nodes-base.merge
• Coden8n-nodes-base.code

From the author

This n8n template demonstrates how to calculate the evaluation metric "Similarity" which in this scenario, measures the consistency of the agent. The scoring approach is adapted from the open-source evaluations project RAGAS and you can see the source here…

View the original on n8n.io → Check your own copy