retriEVAL: evals for AI systems, as an MCP

Try it live

Run a real eval right here, no signup. Pick a sample (or paste your own input, context, and output), choose a free judge model, and watch retriEVAL score it. Nothing is saved.

1 · Choose a sample

2 · Metric

3 · Judge model

Runs on a free model · nothing is saved · limited per day

Sending to the judge model…

What makes it different

The metrics aren't new; DeepEval and Ragas cover those. The shape is: a standalone, local-first MCP an agent can call mid-workflow, with RAG scored by stage.

stage-separated

Retriever vs generator

contextual_recall flags retrieval misses; faithfulness flags hallucinations. Know which half to fix, not just "the answer is bad."

in your chat

Runs inside Claude

Connect it like any other tool and score outputs, author metrics, and pull up charts by asking. retriEVAL is the scoring engine your workflow reaches for.

bring your own model

Your model, your key

Run free on a local Ollama model, or plug in your own Claude, OpenAI, or any OpenAI-compatible LLM key. No platform account, no data leaving your box.

history

Charts + run history

Every run is saved (file or Supabase). Trend by model, compare runs, drill into the case that regressed, on a dashboard you own.

Or just run evals by chatting

No pipeline required. Connect retriEVAL in Claude or another chat UI, toggle it on, and a manual tester can score outputs and author metrics in plain language. Every result links straight to the run in the dashboard.

Toggle it on, like any connector

Connectors

Web search

retriEVAL

Google Drive

Illustration. Uses the hosted server, since chat UIs connect to remote MCPs.

Evals appear right in the chat

Chat

Score this answer against the refund policy: faithfulness and answer relevancy.

Ran 3 metrics on 12 cases 1 FAILING

faithfulness.92PASS

answer_relevancy.88PASS

contextual_recall.58FAIL

View this run in the dashboard

Quickstart

Self-hosted. Clone it, point it at a judge, run it — your data never leaves your machine.

Install & run

git clone https://github.com/hcarrillo001/retrieval-mcp
cd retrieval-mcp
pip install -r requirements.txt

# pick a judge — or use a local one, below
export ANTHROPIC_API_KEY=sk-ant-...
python server.py

Then just ask

# in the chat — paste your cases inline
"Score these cases with faithfulness:
 [{"input": "...",
   "actual_output": "...",
   "retrieval_context": ["..."]}]"

# nothing is stored, no setup step

Pick your judge: free local Ollama (RETRIEVAL_JUDGE_BACKEND=ollama) to keep everything on your machine, or your Anthropic key, your OpenAI key, or any OpenAI-compatible endpoint via OPENAI_BASE_URL. Full setup, including your own always-on instance with a dashboard, is in the repo README.

See it in action

A full walkthrough: connecting the connector, scoring a RAG answer, and reading the result.

Connect retriEVAL and run an eval

Add the connector, load a golden set, score it, and open the run in the dashboard.

Prefer to try it yourself? Run a live eval in the sandbox — no signup.

How it compares

Honest positioning: not "better metrics," but a different shape: standalone, local, and usable as a tool inside an agent.

	retriEVAL	Hosted eval platforms	Eval libraries
Works as an MCP tool inside an agent	yes	platform-gated	no
Runs local, no account	yes	no	yes
Bring your own model / key (Ollama, Claude, OpenAI, …)	yes	limited	yes
RAG scored by stage, built in	yes	varies	manual
Own dashboard + history	yes	yes	no