CLI-Anything-Web Standards (Phase 4: Review + Publish + Verify)

Quality gate for cli-web-* CLIs. This skill owns the complete Phase 4: independent implementation review, structural quality checklist, publishing, and end-user smoke testing. Nothing ships until this phase passes.

Prerequisites (Hard Gate)

Do NOT start unless:

• All tests pass (100% pass rate from Phase 3)
• TEST.md has both Part 1 (plan) and Part 2 (results)
• All core modules are implemented and functional
• <APP>.md (API map) exists and documents all endpoints

If tests are not passing, invoke the testing skill first.

Site Profile Exceptions

Not all checks apply to every CLI. When evaluating, consider the site profile:

• No-auth sites (public APIs): Skip auth-related checks (auth.py required, auth commands, auth smoke test). Mark as N/A.
• Read-only sites (no write operations): Skip write operation smoke test. Verify reads return real data instead.
• API-key auth sites: auth login takes a key argument, not playwright-cli. auth refresh is not applicable — use auth logout instead.

Mark inapplicable checks as "N/A — [reason]" rather than creating dead-code stubs.

Step 1: Implementation Review (3 Parallel Agents)

Before checking structure or publishing, verify the code actually does the right thing. Tests prove it runs; this step proves it's correct.

Dispatch 3 plugin agents in the same message using the Agent tool:

• traffic-fidelity-reviewer — API coverage (reads <APP>.md + client.py + commands/)
• harness-compliance-reviewer — Code conventions (reads HARNESS.md + all source)
• output-ux-reviewer — User experience (runs --help, checks REPL, validates JSON)

Pass each agent: APP_PATH={app}/agent-harness, APP_NAME={app}, and site profile (auth_type, is_read_only). The agents are defined in the plugin's agents/ directory.

Each agent scores findings on a 0-100 confidence scale. When all 3 return:

1. Filter out findings with confidence < 75 (noise)
2. Categorize remaining findings:
   • Critical (90-100): Bugs, missing endpoints, data loss, auth broken
   • Important (75-89): Wrong fields, incomplete parsing, missing options
   • Minor (75, edge cases): Help text gaps, cosmetic issues
3. Present the review report
4. Fix all Critical issues before proceeding — re-run only the affected agent to verify the fix
5. Fix Important issues (not strictly blocking but strongly recommended)

Gate: Do not proceed to Step 2 until Critical count = 0.

Step 2: Structural Quality Checklist (75 checks)

Run the automated checklist validator first to catch mechanical issues:

python ${CLAUDE_PLUGIN_ROOT}/scripts/validate-checklist.py \
  <app>/agent-harness --app-name <app> --auth-type <auth-type>

This checks ~65 of the 75 items automatically (directory structure, required files, CLI patterns, packaging, code quality, REPL, error handling). Fix any FAIL results before proceeding.

For the remaining ~10 judgment-based checks (documentation quality, error message guidance, fixture realism), review manually per references/quality-checklist.md.

Step 3: Create setup.py and Install

1. Create setup.py with:
   • find_namespace_packages for cli_web.*
   • console_scripts entry point: cli-web-<app>
   • Dependencies: click>=8.0, httpx
   • Optional: extras_require={"browser": ["playwright>=1.40.0"]}
2. Install: pip install -e .
3. Verify: which cli-web-<app>
4. Test help: cli-web-<app> --help

Step 4: End-User Smoke Test (MANDATORY)

Run the automated smoke test first for quick validation:

python ${CLAUDE_PLUGIN_ROOT}/scripts/smoke-test.py cli-web-<app> --auth-type <auth-type>

This checks CLI binary resolution, --help, --version, auth status, and --json output for protocol leaks. Then proceed with manual verification below.

This is the most critical verification step. The agent MUST simulate what a real end user would do after pip install cli-web-<app>. If this fails, the pipeline is NOT complete -- go back and fix the issue.

If no-auth site: Skip steps 5-6 (auth). Go directly to step 7 (READ).

If read-only site: Skip step 8 (WRITE). Verify reads return real data.

5. Authenticate as an end user would:

cli-web-<app> auth login

This uses Python sync_playwright() -- opens a browser, user logs in, cookies saved. This is what end users will run. If this fails, the CLI is broken for end users.

6. Verify auth status shows LIVE VALIDATION OK:

cli-web-<app> auth status

Must show: cookies present, tokens valid. If it shows "expired", "redirect", or any auth failure -- STOP. Fix auth before proceeding.

7. Run a READ operation and verify real data:

cli-web-<app> --json <first-resource> list

This must return real data from the live API -- NOT an error, NOT empty, NOT "auth not configured". Verify the JSON response contains expected fields.

8. Run a WRITE operation and verify it actually worked: This is the step the agent most commonly skips. Reading data is easy -- the real test is whether the CLI can CREATE, UPDATE, or GENERATE something.

# For CRUD apps (Monday, Notion, Jira):
cli-web-<app> --json <resource> create --name "smoke-test-$(date +%s)"
cli-web-<app> --json <resource> list   # verify the created item appears
cli-web-<app> --json <resource> delete --id <id-from-create>

# For generation apps (Suno, Midjourney, NotebookLM audio):
cli-web-<app> --json <resource> generate --prompt "test" --wait
# Verify: JSON response contains a real ID, status=complete, not an error
# If the command has --output, verify the file was downloaded and size > 0

# For search/query apps:
cli-web-<app> --json search "test query"
# Verify: results array is non-empty

If ANY write/generate command fails, the pipeline is NOT complete. Reading a list of existing items only proves auth works -- it does NOT prove the CLI can actually do useful work. The whole point is to CREATE things, not just read them.

9. Only after steps 5-8 ALL pass, declare the pipeline complete.

Smoke Test Checklist

• auth login works (Python playwright, API key, or N/A for no-auth)
• auth status shows valid (or N/A for no-auth)
• At least one READ returns real data
• At least one WRITE/CREATE/GENERATE succeeds (or N/A for read-only)
• The CLI works standalone -- no debug Chrome, no port 9222, no MCP
• Output sanity: no raw protocol data leaks in --json output (see below)

Output Sanity

Run every command with --json and check for raw protocol leaks (wrb.fr, af.httprm, empty [], null required fields). See methodology/SKILL.md "Mandatory Smoke Check" for the full red flags list.

#1 gap to watch for: Agent runs list (GET with auth — easy), declares done, but never tests create/generate (POST with CSRF, encoding). Always test at least one write.

Post-Smoke-Test: Generate Skill + Update README (Parallel)

After smoke tests pass, these tasks remain — all independent, dispatch in parallel:

┌─ Agent 1: Generate Claude Skill (.claude/skills/<app>-cli/SKILL.md)
│           ALSO copy to cli_web/<app>/skills/SKILL.md (package-portable)
├─ Agent 2: Update repository README.md (add CLI to examples table)
├─ Agent 3: Write/update cli_web/<app>/README.md (package docs)
├─ Agent 4: Update registry.json + CLAUDE.md Generated CLIs table
└─ Agent 5: Add CLI to CI test matrix (.github/workflows/tests.yml)
│           + Add entry to CHANGELOG.md under [Unreleased]
All are independent — launch in one message with run_in_background: true

Use the templates at cli-anything-web-plugin/templates/ as the canonical structure for SKILL.md and README.md — fill in the {{placeholders}} with actual CLI data from <app> --help and <APP>.md.

Generate Claude Skill

Goal: Create a project-local Claude skill so that Claude can use this CLI automatically in future conversations — no manual lookup required.

IMPORTANT: The skill must exist in TWO locations:

1. .claude/skills/<app>-cli/SKILL.md — for Claude Code discovery (project-level)
2. <app>/agent-harness/cli_web/<app>/skills/SKILL.md — portable with pip install (included via package_data in setup.py)

Create the skill once, then copy it to both locations.

Step 1: Find the .claude directory

Create <git-root>/.claude/skills/<app>-cli/SKILL.md:

1. Read the CLI's README and run cli-web-<app> --help + <resource> --help
2. Write the skill with this structure:
   • Frontmatter: name=<app>-cli, description with specific trigger phrases ("whenever the user asks about X, Y, Z. Always prefer cli-web- over manually fetching the website.")
   • Quick Start: 2-3 most common commands with --json
   • Commands: each command group with key options and output fields
   • Agent Patterns: piped command examples for common tasks
   • Notes: auth setup, rate limits, known limitations
3. Use existing skills (e.g., notebooklm-cli, futbin-cli) as reference examples

Update Repository README

Add the new CLI to the examples table in README.md (CLI name, website, protocol, auth type, description) and add a quick-start example in the "Try Them" section.

Update registry.json and CLAUDE.md

Add the new CLI to registry.json at the repo root:

{
  "name": "cli-web-<app>",
  "website": "<website>",
  "protocol": "<detected protocol>",
  "auth": "<auth type>",
  "directory": "<app>/agent-harness",
  "namespace": "cli_web.<app>",
  "commands": ["<cmd1>", "<cmd2>", ...],
  "install": "pip install -e <app>/agent-harness"
}

Also add to the Generated CLIs table in CLAUDE.md.

Pipeline Complete

The pipeline is NOT done until ALL of these are checked:

Smoke Tests

• Auth works (login + status, or N/A for no-auth)
• At least one READ returns real data
• At least one WRITE succeeds (or N/A for read-only)

Skills (TWO copies)

• .claude/skills/<app>-cli/SKILL.md exists (Claude Code discovery)
• cli_web/<app>/skills/SKILL.md exists (portable with pip install)
• Used cli-anything-web-plugin/templates/SKILL.md.template as starting point

Package

• setup.py has package_data={"": ["skills/*.md", "*.md"]}
• __main__.py exists for python -m cli_web.<app> support

Documentation

• cli_web/<app>/README.md exists (used templates/README.md.template)
• <APP>.md API map exists
• tests/TEST.md has Part 1 (plan) + Part 2 (results)

Repo-Level Updates

• README.md — new row in examples table + "Try them" section
• README.md — badge count updated (CLIs_generated-N and N_CLIs hero badge)
• CLAUDE.md — new row in Generated CLIs table
• registry.json — entry with name, website, protocol, auth, commands, install
• docs/registry/index.html — entry added to JS data array with correct category
• CHANGELOG.md — entry added under [Unreleased] → Added
• .github/workflows/tests.yml — new CLI added to CI test matrix (see below)

CI Test Matrix Update (MANDATORY)

Every new CLI MUST be added to .github/workflows/tests.yml so unit tests run on every push/PR. Do both steps — missing either blocks merges.

Step 1: Add to test matrix in .github/workflows/tests.yml:

- { name: <app>, dir: <app>/agent-harness, pkg: <app_underscore> }

Where <app_underscore> replaces hyphens with underscores (e.g., gh-trending → gh_trending).

Step 2: Add to branch protection required checks so PRs require the new check:

# Get current checks, append the new one, update
gh api repos/<owner>/<repo>/branches/main/protection/required_status_checks \
  -X PATCH --input - <<EOF
{"strict": true, "contexts": [...existing..., "<app>"]}
EOF

Verify the entry runs: python -m pytest <dir>/cli_web/<pkg>/tests/test_core.py -v

All key rules (naming, auth, --json, REPL, rate limits) are defined in HARNESS.md "Critical Rules" and CLAUDE.md "Critical Conventions".

Integration

Related

• testing skill -- Phase 3 test planning/writing/documentation
• methodology skill -- Phase 2 analyze/design/implement
• capture skill -- Phase 1 traffic recording
• /cli-anything-web:validate -- Command to run the full 75-check validation