Test Evidence Review

Test Evidence Review

What Is This

The Test Evidence Review skill (test-evidence-review) is an advanced quality assurance tool designed for the Happycapy Skills platform. Unlike basic existence checks that only confirm if test files are present and passing, this skill performs a comprehensive quality review of both automated test files and manual evidence documents. Its primary function is to evaluate the thoroughness and reliability of the test evidence associated with a software story, sprint, or system. The skill analyzes assertion coverage, edge case handling, naming conventions, and the completeness of manual sign-offs. It then generates a clear verdict for each story: ADEQUATE, INCOMPLETE, or MISSING. The review process can be invoked before QA sign-off, on demand when test quality is uncertain, or as part of broader milestone audits.

Why Use It

Relying solely on the existence and passing status of test files can create a false sense of security. Test files that pass may still omit critical edge cases or lack comprehensive coverage, leading to undetected defects. Similarly, manual evidence documents may be present but incomplete, lacking the required approvals for closure.

The Test Evidence Review skill addresses these gaps by:

• Ensuring that all critical behaviors are adequately tested and documented.
• Identifying missing or insufficient assertion coverage in automated tests.
• Verifying that edge cases are explicitly handled.
• Checking for proper and descriptive test naming conventions.
• Auditing manual evidence documents for completeness, including necessary sign-offs.
• Producing actionable summary reports, facilitating smoother QA hand-offs and reducing the risk of post-release defects.

By automating these reviews, teams maintain a consistently high standard of test evidence, which is essential for regulated environments, complex integrations, and high-stakes releases.

How to Use It

The skill is invoked using the /test-evidence-review command, followed by an argument specifying the review scope. The accepted arguments are:

• [story-path]: Review evidence for a single story.
• sprint: Review evidence for all stories in the current sprint.
• [system-name]: Review evidence for all stories within a specified system or epic.
• No argument: The skill will prompt you to choose the scope interactively ("Single story", "Current sprint", or "A system").

Example Usage:

/test-evidence-review stories/authentication/reset-password

This command reviews the test and evidence files for the "reset-password" story within the authentication module.

/test-evidence-review sprint

This command reviews all stories in the current sprint.

Workflow:

1. Parse Arguments: The skill interprets the command and determines the scope based on your input.
2. Load Stories in Scope: It identifies the relevant stories and associated test/evidence files.
3. Analyze Test Files: For each story, it inspects automated tests for:
   • Assertion coverage
   • Presence of meaningful edge case tests
   • Descriptive and consistent test naming
4. Analyze Manual Evidence: The skill checks for the existence and completeness of manual evidence documents, including required sign-offs.
5. Generate Report: It produces a summary report (in conversation) with a verdict (ADEQUATE, INCOMPLETE, MISSING) per story. Optionally, it can write a detailed markdown report to a file such as production/qa/evidence-review-[date].md.

Sample Output:

Story: authentication/reset-password
Automated Tests: ADEQUATE (covers all scenarios, edge cases included)
Manual Evidence: INCOMPLETE (missing sign-off from security lead)
Verdict: INCOMPLETE

When to Use It

• Before QA hand-off sign-off: Run this skill during the /team-qa Phase 5 to ensure all stories meet quality standards before transitioning to QA.
• On demand: Use it whenever there is doubt about the quality or completeness of test evidence for a specific story.
• Milestone review: Integrate this review as part of milestone or release audits, especially for logic-heavy or integration-focused stories.
• Regulatory or compliance checks: Use it in projects with strict documentation or sign-off requirements.

Important Notes

• The skill's assessment is qualitative. It does not only check for the presence of files but evaluates their actual content and coverage.
• The verdicts are story-specific, allowing targeted remediation.
• Output includes both a summary in the chat and, optionally, a detailed markdown report saved to the repository.
• This is not a replacement for human QA but an accelerator and enhancer of your existing review processes.
• Requires access permissions to read, glob, grep, and write files in the relevant directories.
• If no argument is provided, the skill will prompt for the desired scope, ensuring user intent is always clear.

By integrating Test Evidence Review into your development workflow, you elevate the standard of your software quality assurance, reduce risk, and streamline hand-offs between development and QA teams.