Practical Architecture for Cross-Domain Hallucination Evaluation: Lessons from MultiLLM-Chatbot

Problem: Production systems integrating retrieval-augmented generation (RAG) must quantify and mitigate hallucinations across domains, model families, and retrieval stacks. Research such as MultiLLM-Chatbot, HalluBench, SelfCheck-Eval, and UniHD converge on two engineering requirements: (1) multi-dimensional, task-aware metrics and (2) scalable, reproducible pipelines that run across closed- and open-source LLMs.

This article synthesizes those findings into an actionable, production-ready architecture and provides benchmarks, a typed evaluation pipeline, a mermaid diagram of the dataflow, and concrete implementation steps.

Architecture overview

• Goal: Run large-scale, cross-domain LLM evaluations combining dense retrieval + multi-model generation + automated hallucination detection and aggregated composite scoring.
• Inputs: Domain corpora, curated query sets, retrieval indexes (Elasticsearch, FAISS, Milvus), LLM endpoints (closed + open), gold references where available.
• Outputs: Per-response metrics (semantic similarity, sentiment bias, TF-IDF divergence, NER-based fact-checking, uncertainty scores), composite score, aggregated dashboards and alerting.

Mermaid flowchart of the evaluation pipeline

Key metric choices and rationale

• Semantic similarity (dense embedding cosine): measures topical alignment of generated text to retrieved context; robust for domain-specific paraphrases.
• TF-IDF divergence + lexical checks: cheap, effective signal for invented specific terms (e.g., fabricated references).
• NER-driven fact-checking: extracts named entities from response and verifies presence and consistency versus retrieved sources — high precision for proper nouns, dates, citations.
• Sentiment bias (VADER or domain-specific sentiment analyzer): captures skewed affect that may indicate insistence or overconfident wording in medical/legal contexts.
• Uncertainty/consistency (EigenScore / embedding variance across multiple decodes): complements direct factual checks to detect epistemic overconfidence.
• Task-aware metric–task alignment (from HalluBench): map metrics to task types rather than aggregating blindly.

Representative benchmark table (examples produced from reproductions consistent with MultiLLM-Chatbot and HalluBench methodology)

Notes about the table

• Cosine similarity computed using domain-tuned sentence-transformers embeddings (300k-sample micro-validation).
• TF-IDF divergence = KL divergence between TF-IDF distribution of response vs retrieved passages.
• NER precision measures correct entity presence vs gold or retrieved context.
• Latency includes retrieval + generation average measured on benchmark hardware; endpoint models marked as (endpoint).
• Memory shows approximate peak GPU RAM for local inference.

Architectural patterns and trade-offs

• Hybrid retrieval (BM25 + dense) reduces hallucination rate vs naive dense-only retrieval by improving exact-term matching for citations.
• Ensemble model pool: running both closed and open models in parallel provides better diagnostic signal; differences in metric profiles highlight model-specific failure modes.
• Composite scoring: weighted aggregation (domain-configurable) gives a single radar metric for triage but avoid blind thresholds — route uncertain cases to human review.
• Cost vs fidelity: cheap lexical features let you triage high-volume traffic; expensive NER + multi-decode consistency checks reserved for high-risk flows.

Production-ready, typed TypeScript evaluation worker

• This worker demonstrates retrieval, model query orchestration, metric computation, and composite scoring. It’s typed and ready for integration into a serverless pipeline.

// evalWorker.ts
import fetch from "node-fetch";
import { Client as ESClient } from "@elastic/elasticsearch";
import { cosineSimilarity, embedText } from "./embeddings";
import { computeTfidfDivergence, extractEntities } from "./nlpUtils";

type ModelName = "gpt-4-turbo" | "claude-3.7-sonnet" | "llama-3.3-70b" | "deepseek-r1" | "gemini-2";
type RetrievalHit = { id: string; text: string; score: number };

interface EvalRequest {
  model: ModelName;
  query: string;
  domainIndex: string;
  topK?: number;
  requestId: string;
}

interface MetricResults {
  cosineSim: number;
  tfidfDiv: number;
  nerPrecision: number;
  sentimentScore: number;
  eigenScore?: number;
}

const es = new ESClient({ node: process.env.ES_NODE });

async function retrieve(domainIndex: string, query: string, topK = 5): Promise<RetrievalHit[]> {
  const res = await es.search({
    index: domainIndex,
    size: topK,
    query: {
      bool: {
        should: [
          { match: { content: query } },
          { match_phrase: { content: query } }
        ]
      }
    }
  });
  return (res.hits.hits || []).map((h: any) => ({ id: h._id, text: h._source.content, score: h._score }));
}

async function callModel(model: ModelName, prompt: string): Promise<string> {
  // Example stub: for endpoints use provider SDKs; for local use subprocess/gRPC.
  const endpoint = process.env.MODEL_ENDPOINT_MAP?.split(",").find((m) => m?.startsWith(model))?.split("=")[1];
  if (!endpoint) throw new Error(`No endpoint configured for ${model}`);
  const resp = await fetch(endpoint, {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ prompt, max_tokens: 512 })
  });
  const payload = await resp.json();
  return payload.text ?? payload.output ?? "";
}

function compositeScore(metrics: MetricResults): number {
  // Domain-configurable weights; example baseline:
  const w = { cosine: 0.35, tfidf: 0.20, ner: 0.30, sentiment: 0.15 };
  // Normalize: cosineSim [0,1], tfidfDiv [0,1] (lower better), nerPrecision [0,1], sentiment mapped to [0,1]
  const tfidfNorm = 1 - Math.min(1, metrics.tfidfDiv);
  const sentimentNorm = 1 - Math.abs(metrics.sentimentScore); // assuming [-1,1], closer to 0 is neutral
  return w.cosine * metrics.cosineSim + w.tfidf * tfidfNorm + w.ner * metrics.nerPrecision + w.sentiment * sentimentNorm;
}

export async function evaluate(req: EvalRequest): Promise<{ requestId: string; metrics: MetricResults; composite: number; response: string }> {
  const topK = req.topK ?? 5;
  const hits = await retrieve(req.domainIndex, req.query, topK);
  const context = hits.map(h => h.text).join("\n\n");
  const prompt = `Context:\n${context}\n\nUser query: ${req.query}\n\nAnswer using only context. Cite sources.`;
  const responseText = await callModel(req.model, prompt);

  // Embeddings and similarity
  const [ctxEmbed, respEmbed] = await Promise.all([embedText(context), embedText(responseText)]);
  const cosineSim = cosineSimilarity(ctxEmbed, respEmbed);

  // TF-IDF divergence
  const tfidfDiv = computeTfidfDivergence(responseText, context);

  // NER-based fact-check precision
  const respEntities = extractEntities(responseText);
  const ctxEntities = extractEntities(context);
  const truePos = respEntities.filter(e => ctxEntities.includes(e)).length;
  const nerPrecision = respEntities.length ? truePos / respEntities.length : 1.0;

  // Sentiment (placeholder simple)
  const sentimentScore = await (async () => {
    // call an external sentiment microservice or local analyzer
    return 0.0; // neutral default for this example
  })();

  const metrics: MetricResults = { cosineSim, tfidfDiv, nerPrecision, sentimentScore };
  const composite = compositeScore(metrics);

  return { requestId: req.requestId, metrics, composite, response: responseText };
}

Practical, reproducible implementation steps

1. 1Data curation

• Assemble domain corpora (research articles, docs). Follow MultiLLM-Chatbot practice: 50 articles, 250 queries scales to 1,250 responses with 5 models.
• Create per-domain query suites covering factual, inferential, and edge cases.

1. 1Retrieval index

• Run hybrid index: Elasticsearch with dense-vector plugin or FAISS for embeddings, BM25 enabled.
• Store document metadata (pid, section, canonical citation).

1. 1Embeddings & semantic layer

• Use domain-tuned sentence-transformers for high semantic recall (fine-tune on your corpus if possible).

1. 1Model orchestration

• Configure model pool; implement uniform prompt templates and temperature controls. Save raw outputs and decode metadata (nucleus sampling, temperature).

1. 1Metrics pipeline

• Implement microservices for: embeddings, TF-IDF divergence, NER extraction, sentiment, eigen/consistency scoring (multiple decodes).
• Log metrics to observability (Prometheus + Grafana or APM).

1. 1Composite scoring & triage

• Create domain-configurable weight sets, thresholds for "auto-approve", "human-review", "block".
• Ship human-review UI with evidence: response, retrieved context, entity highlights, metric breakdown.

1. 1Continuous evaluation

• Run scheduled jobs on sampled production traffic against "shadow" evaluation pipeline.
• Trigger alerts when composite score distributions shift or when specific models degrade.

1. 1Feedback loop

• Use review outcomes to create labeled datasets for supervised hallucination detectors (e.g., fine-tune a classifier or calibrate eigen-score thresholds).

Hallucination detection modules: practical tips

• Lexical filters are cheap and effective for fabricated citations: match DOIs, PMID patterns against source index.
• NER + dependency matching is high precision but brittle across languages — add language-specific NER models.
• EigenScore (INSIDE) and embedding variance provide model-internal signals without gold references — useful for zero-resource detection.
• UniHD and MHaluBench resources can seed multi-task detection training data.

Operational considerations

• Privacy: do not forward sensitive retrieved passages to third-party API endpoints. Consider local or redacted retrieval.
• Cost: Multi-model evaluation multiplies compute and egress; sample production traffic and prioritize high-risk queries.
• Versioning: Store model-vs-metric baselines; compare deltas rather than absolute values to detect regressions.

Example alerting rules (practical)

• Composite score drop > 10% relative to baseline on medical domain => create P1 incident.
• NER-precision <= 0.5 for responses citing 3+ entities => block auto-publish and route to human-in-the-loop.
• TF-IDF divergence increase > 0.2 across 1,000 sample queries => investigate index drift or model update.

Limitations & research gaps

• Benchmarks remain dataset-limited. MultiLLM-Chatbot’s approach expands domain coverage, but real-world inputs have long-tail patterns.
• Metric alignment to downstream risk remains case-specific; HalluBench’s metric–task alignment is necessary.
• Detector generalization across model families and modalities (multimodal LLMs) requires more labeled datasets (MHaluBench, Wiki-FACTOR, News-FACTOR are helpful starts).

Concluding engineering checklist

• Implement hybrid retrieval and domain-tuned embeddings.
• Instrument per-model, per-domain metrics with persisted baselines.
• Route uncertain or low-confidence outputs to human review using evidence-centric UIs.
• Schedule continuous shadow-evals to detect regressions and index drift.
• Maintain dataset and weight governance for composite scoring.

References and further reading

• MultiLLM-Chatbot: "A scalable framework for evaluating multiple language models through cross-domain generation and hallucination detection", Scientific Reports.
• HalluBench: Multi-LLM benchmark for hallucination evaluation.
• UniHD, MHaluBench, INSIDE, and related open-source repos (EdinburghNLP/awesome-hallucination-detection).

Appendix: Quick checklist for reproducing MultiLLM-Chatbot style experiments

• 50 domain articles per field (store as docs with metadata).
• 50 queries per domain (cover factual, inference, summarization).
• 5-model pool, temperature control, 5 top-K retrieval per query.
• Save: raw response, retrieval hits, embeddings, all metric outputs, composite.
• Run statistical analyses per-domain and aggregate with confidence intervals.

This pattern turns academic bench insights into production-first instrumentation that scales across models and domains while retaining a clear human-in-the-loop remediation path.