Design an AI Evaluation Platform

The flow

1. 1.Freeze manifest
2. 2.Queue cases
3. 3.Run isolated candidates
4. 4.Score and review
5. 5.Compare slices
6. 6.Gate release

Architecture and boundaries

Design assumptions

These values are example sizing inputs, not measurements or capacity guarantees.

Version the experiment contract

Capture immutable dataset version, application commit, prompt revision, retrieval snapshot, model identifier and scorer configuration in each run. A seed alone does not guarantee identical hosted-model output. Store outputs and provenance so a reviewer can inspect what actually ran.

Separate development examples from held-out release cases. Track consent, access and retention for production-derived data. A larger dataset is not automatically representative: tag language, tenant, risk and task difficulty, and review coverage gaps.

Isolate workers and bound the run

Persist a run manifest before queueing cases. Workers claim cases with expiring leases, enforce deadlines and write results keyed by run, case and attempt. Keep failed and timed-out cases visible; excluding them can inflate quality scores.

Stub external write tools or use an isolated test environment. Limit provider concurrency, calls and spending per run. Treat candidate output as untrusted input to the scorer; do not let an answer instruct the judge to change its rubric. Repeated jobs must not publish duplicate reports.

Compare evidence before promotion

Use schema and policy checks for objective constraints; use calibrated human review for subjective quality and a versioned judge only where appropriate. Compare the same cases in baseline and candidate runs. Report sample counts and uncertainty, not just one average score.

Assume 1,000 cases, two candidates and three repetitions: 6,000 application invocations. At 20 concurrent slots and 4 seconds mean duration, ideal application time is 1,200 seconds before judge calls, throttling and failures. Gate on critical violations, slice regressions and operational budgets; a missing or incomplete report must not count as a pass.

Try it

{
  "datasetVersion": "support-heldout-v3",
  "candidates": [
    "baseline",
    "candidate"
  ],
  "repetitions": 3,
  "maxConcurrentCases": 20,
  "caseTimeoutSeconds": 30,
  "requiredCompletedCases": 1000,
  "blockOnCriticalPolicyFailure": true
}

Illustrative design contract, not a deployable service configuration. Validate limits with load and failure tests.

Interview questions

Official references

• MLflow — evaluation dataset concepts
• MLflow — LLM and agent evaluation

Continue learning

• LLM Evaluation →
• Agent Evaluation & Reliability →