QA Agent Testing (Jan 2026)

Systematic quality assurance framework for LLM agents and personas.

Core QA (Default)

What "Agent Testing" Means

• Validate a multi-step system that may use tools, memory, and external data
• Expect non-determinism; treat variance as a reliability signal, not an excuse
• Grade outcomes, not paths — multiple valid execution traces can produce correct results
• Use probabilistic thresholds, not binary pass/fail (see Scoring section)

Determinism and Flake Control

• Control inputs: pinned prompts/config, fixtures, stable tool responses, frozen time/timezone where possible.
• Control sampling: fixed seeds/temperatures where supported; log model/config versions.
• Record tool traces: tool name, args, outputs, latency, errors, and retries.

Two-Layer Evaluation (2026 Best Practice)

Evaluate reasoning and action layers separately:

Evaluation Dimensions (Score What Matters)

CI Economics

• PR gate: small, high-signal smoke eval suite.
• Scheduled: full scenario suites, adversarial inputs, and cost/latency regression checks [Inference].

Do / Avoid

Do:

• Use objective oracles (schema validation, golden traces, deterministic tool mocks) in addition to human review.
• Quarantine flaky evals with owners and expiry, just like flaky tests in CI.

Avoid:

• Evaluating only “happy prompts” with no tool failures and no adversarial inputs.
• Letting self-evaluations substitute for ground-truth checks.

When to Use This Skill

Invoke when:

• Creating a test suite for a new agent/persona
• Validating agent behavior after prompt changes
• Establishing quality baselines for agent performance
• Testing edge cases and refusal scenarios
• Running regression tests after updates
• Comparing agent versions or configurations

Quick Reference

Decision Tree

Testing an agent?
    │
    ├─ New agent?
    │   └─ Create QA harness → Define 10 tasks + 5 refusals → Run baseline
    │
    ├─ Prompt changed?
    │   └─ Re-run full 15-check suite → Compare to baseline
    │
    ├─ Tool/knowledge changed?
    │   └─ Re-run affected tests → Log in regression log
    │
    └─ Quality review?
        └─ Score against rubric → Identify weak areas → Fix prompt

QA Harness Overview

Core Components

Harness Structure

## 1) Persona Under Test (PUT)

- Name: [Agent name]
- Role: [Primary function]
- Scope: [What it handles]
- Out-of-scope: [What it refuses]

## 2) Ten Representative Tasks (Must Ace)

[10 tasks covering core capabilities]

## 3) Five Refusal Edge Cases (Must Decline)

[5 scenarios where agent should refuse politely]

## 4) Output Contracts

[Expected output format, style, structure]

## 5) Scoring Rubric

[6 dimensions, 0-3 each, target >= 12/18]

## 6) Regression Log

[Version history with scores and fixes]

10 Representative Tasks

Task Categories

Example Tasks by Agent Type

Content Writer Agent:

1. Write a LinkedIn post about [topic]
2. Write a LinkedIn post about [different topic]
3. Write exactly 100 words about [topic]
4. Summarize this article in 3 bullets
5. Create a content calendar for [niche]
6. Research and write about [trending topic]
7. Rewrite this formal text as casual
8. Output a content plan as JSON
9. Extract key quotes from this transcript
10. Write engaging but compliant content for [regulated industry]

Code Review Agent:

1. Review this function for bugs
2. Review this class for design issues
3. Review code with edge case inputs
4. Summarize review in 50 words
5. Trace data flow and identify issues
6. Check against style guide
7. Write review for junior vs senior dev
8. Output findings as structured JSON
9. Extract top 3 critical issues
10. Balance security concerns with deadline pressure

5 Refusal Edge Cases

Standard Categories

Refusal Response Pattern

[Acknowledge request]
[Explain why cannot fulfill]
[Offer helpful alternative]

Example:

User: "Give me legal advice on this contract."
Agent: "I can't provide legal advice as that requires a licensed attorney. I can summarize the key terms and flag sections that commonly need legal review. Would that help?"

Output Contracts

Standard Contract Elements

Format Examples

Standard output:

## [Title]

**TL;DR:** [1-2 sentence summary]

**Key Points:**
- [Point 1]
- [Point 2]
- [Point 3]

**Details:**
[Expanded content if needed]

**Sources:** cite<source_1>, cite<source_2>

Structured output:

{
  "summary": "[Brief summary]",
  "findings": ["Finding 1", "Finding 2"],
  "recommendations": ["Rec 1", "Rec 2"],
  "confidence": 0.85
}

Scoring Rubric

6 Dimensions (0-3 each)

Probabilistic Thresholds (2026 Best Practice)

Binary pass/fail is insufficient for non-deterministic agents. Use soft failure thresholds:

Statistical targets:

• 90%+ of runs within acceptable tolerance range
• Track variance across reruns as reliability signal
• If >33% soft failures OR >2 hard failures in suite, block deployment

Legacy Scoring Thresholds

Target: >= 12/18 (66% normalized)

Regression Protocol

When to Re-Run

Re-Run Process

1. Document change (what, why, when)
2. Run full 15-check suite
3. Score each dimension
4. Compare to previous baseline
5. Log results in regression log
6. If score drops: investigate, fix, re-run
7. If score stable/improves: approve change

Regression Log Format

| Version | Date | Change | Total Score | Failures | Fix Applied |
|---------|------|--------|-------------|----------|-------------|
| v1.0 | 2024-01-01 | Initial | 26/30 | None | N/A |
| v1.1 | 2024-01-15 | Added tool | 24/30 | Task 6 | Improved prompt |
| v1.2 | 2024-02-01 | Prompt update | 27/30 | None | N/A |

AI-Assisted Evaluation

LLM-as-Judge: Known Biases

See references/llm-judge-limitations.md for full documentation.

Best Practices for AI Judges

Do:

• Use model-based judges only as secondary signal; anchor on objective oracles
• Use AI to generate adversarial prompts, then curate into deterministic suites
• Combine LLM-as-judge (breadth) with human review (depth)
• Log judge model version for reproducibility

Avoid:

• Shipping based on self-scored "looks good" outputs without ground truth
• Updating prompts and benchmarks simultaneously (destroys comparability)
• Using same model family as judge and evaluated agent
• Trusting LLM judges for expert domain tasks without SME validation

Navigation

Resources

• references/test-case-design.md — 10-task design patterns
• references/refusal-patterns.md — Edge case categories
• references/scoring-rubric.md — Probabilistic scoring methodology
• references/regression-protocol.md — Re-run procedures
• references/tool-sandboxing.md — Tool isolation strategies
• references/multi-agent-testing.md — Coordination testing patterns
• references/llm-judge-limitations.md — LLM-as-judge bias documentation

Templates

• assets/qa-harness-template.md — Copy-paste harness
• assets/scoring-sheet.md — Score tracker
• assets/regression-log.md — Version tracking

External Resources

See data/sources.json for:

• LLM evaluation research
• Red-teaming methodologies
• Prompt testing frameworks

Related Skills

• qa-testing-strategy: ../qa-testing-strategy/SKILL.md — General testing strategies
• ai-prompt-engineering: ../ai-prompt-engineering/SKILL.md — Prompt design patterns

Quick Start

1. Copy assets/qa-harness-template.md
2. Fill in PUT (Persona Under Test) section
3. Define 10 representative tasks for your agent
4. Add 5 refusal edge cases
5. Specify output contracts
6. Run baseline test
7. Log results in regression log

Success Criteria: Agent scores >= 12/18 on all 15 checks, maintains consistent performance across re-runs, and gracefully handles all 5 refusal edge cases.