pig/pig-farm-controller
1
0
Fork 0
Code Issues 12 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
426ae41f54d154aa0ec9b4338751654a8cae1ca5
pig-farm-controller/bmad/bmm/workflows/testarch/trace/checklist.md
huang 426ae41f54 bmad初始化
2025-11-01 19:22:39 +08:00

21 KiB
Raw Blame History

Requirements Traceability & Gate Decision - Validation Checklist

Workflow: testarch-trace Purpose: Ensure complete traceability matrix with actionable gap analysis AND make deployment readiness decision (PASS/CONCERNS/FAIL/WAIVED)

This checklist covers two sequential phases:

  • PHASE 1: Requirements Traceability (always executed)
  • PHASE 2: Quality Gate Decision (executed if enable_gate_decision: true)

PHASE 1: REQUIREMENTS TRACEABILITY

Prerequisites Validation

  • Acceptance criteria are available (from story file OR inline)
  • Test suite exists (or gaps are acknowledged and documented)
  • Test directory path is correct (test_dir variable)
  • Story file is accessible (if using BMad mode)
  • Knowledge base is loaded (test-priorities, traceability, risk-governance)

Context Loading

  • Story file read successfully (if applicable)
  • Acceptance criteria extracted correctly
  • Story ID identified (e.g., 1.3)
  • test-design.md loaded (if available)
  • tech-spec.md loaded (if available)
  • PRD.md loaded (if available)
  • Relevant knowledge fragments loaded from tea-index.csv

Test Discovery and Cataloging

  • Tests auto-discovered using multiple strategies (test IDs, describe blocks, file paths)
  • Tests categorized by level (E2E, API, Component, Unit)
  • Test metadata extracted:
    • Test IDs (e.g., 1.3-E2E-001)
    • Describe/context blocks
    • It blocks (individual test cases)
    • Given-When-Then structure (if BDD)
    • Priority markers (P0/P1/P2/P3)
  • All relevant test files found (no tests missed due to naming conventions)

Criteria-to-Test Mapping

  • Each acceptance criterion mapped to tests (or marked as NONE)
  • Explicit references found (test IDs, describe blocks mentioning criterion)
  • Test level documented (E2E, API, Component, Unit)
  • Given-When-Then narrative verified for alignment
  • Traceability matrix table generated:
    • Criterion ID
    • Description
    • Test ID
    • Test File
    • Test Level
    • Coverage Status

Coverage Classification

  • Coverage status classified for each criterion:
    • FULL - All scenarios validated at appropriate level(s)
    • PARTIAL - Some coverage but missing edge cases or levels
    • NONE - No test coverage at any level
    • UNIT-ONLY - Only unit tests (missing integration/E2E validation)
    • INTEGRATION-ONLY - Only API/Component tests (missing unit confidence)
  • Classification justifications provided
  • Edge cases considered in FULL vs PARTIAL determination

Duplicate Coverage Detection

  • Duplicate coverage checked across test levels
  • Acceptable overlap identified (defense in depth for critical paths)
  • Unacceptable duplication flagged (same validation at multiple levels)
  • Recommendations provided for consolidation
  • Selective testing principles applied

Gap Analysis

  • Coverage gaps identified:
    • Criteria with NONE status
    • Criteria with PARTIAL status
    • Criteria with UNIT-ONLY status
    • Criteria with INTEGRATION-ONLY status
  • Gaps prioritized by risk level using test-priorities framework:
    • CRITICAL - P0 criteria without FULL coverage (BLOCKER)
    • HIGH - P1 criteria without FULL coverage (PR blocker)
    • MEDIUM - P2 criteria without FULL coverage (nightly gap)
    • LOW - P3 criteria without FULL coverage (acceptable)
  • Specific test recommendations provided for each gap:
    • Suggested test level (E2E, API, Component, Unit)
    • Test description (Given-When-Then)
    • Recommended test ID (e.g., 1.3-E2E-004)
    • Explanation of why test is needed

Coverage Metrics

  • Overall coverage percentage calculated (FULL coverage / total criteria)
  • P0 coverage percentage calculated
  • P1 coverage percentage calculated
  • P2 coverage percentage calculated (if applicable)
  • Coverage by level calculated:
    • E2E coverage %
    • API coverage %
    • Component coverage %
    • Unit coverage %

Test Quality Verification

For each mapped test, verify:

  • Explicit assertions are present (not hidden in helpers)
  • Test follows Given-When-Then structure
  • No hard waits or sleeps (deterministic waiting only)
  • Self-cleaning (test cleans up its data)
  • File size < 300 lines
  • Test duration < 90 seconds

Quality issues flagged:

  • BLOCKER issues identified (missing assertions, hard waits, flaky patterns)
  • WARNING issues identified (large files, slow tests, unclear structure)
  • INFO issues identified (style inconsistencies, missing documentation)

Knowledge fragments referenced:

  • test-quality.md for Definition of Done
  • fixture-architecture.md for self-cleaning patterns
  • network-first.md for Playwright best practices
  • data-factories.md for test data patterns

Phase 1 Deliverables Generated

Traceability Matrix Markdown

  • File created at {output_folder}/traceability-matrix.md
  • Template from trace-template.md used
  • Full mapping table included
  • Coverage status section included
  • Gap analysis section included
  • Quality assessment section included
  • Recommendations section included

Coverage Badge/Metric (if enabled)

  • Badge markdown generated
  • Metrics exported to JSON for CI/CD integration

Updated Story File (if enabled)

  • "Traceability" section added to story markdown
  • Link to traceability matrix included
  • Coverage summary included

Phase 1 Quality Assurance

Accuracy Checks

  • All acceptance criteria accounted for (none skipped)
  • Test IDs correctly formatted (e.g., 1.3-E2E-001)
  • File paths are correct and accessible
  • Coverage percentages calculated correctly
  • No false positives (tests incorrectly mapped to criteria)
  • No false negatives (existing tests missed in mapping)

Completeness Checks

  • All test levels considered (E2E, API, Component, Unit)
  • All priorities considered (P0, P1, P2, P3)
  • All coverage statuses used appropriately (FULL, PARTIAL, NONE, UNIT-ONLY, INTEGRATION-ONLY)
  • All gaps have recommendations
  • All quality issues have severity and remediation guidance

Actionability Checks

  • Recommendations are specific (not generic)
  • Test IDs suggested for new tests
  • Given-When-Then provided for recommended tests
  • Impact explained for each gap
  • Priorities clear (CRITICAL, HIGH, MEDIUM, LOW)

Phase 1 Documentation

  • Traceability matrix is readable and well-formatted
  • Tables render correctly in markdown
  • Code blocks have proper syntax highlighting
  • Links are valid and accessible
  • Recommendations are clear and prioritized

PHASE 2: QUALITY GATE DECISION

Note: Phase 2 executes only if enable_gate_decision: true in workflow.yaml

Prerequisites

Evidence Gathering

  • Test execution results obtained (CI/CD pipeline, test framework reports)
  • Story/epic/release file identified and read
  • Test design document discovered or explicitly provided (if available)
  • Traceability matrix discovered or explicitly provided (available from Phase 1)
  • NFR assessment discovered or explicitly provided (if available)
  • Code coverage report discovered or explicitly provided (if available)
  • Burn-in results discovered or explicitly provided (if available)

Evidence Validation

  • Evidence freshness validated (warn if >7 days old, recommend re-running workflows)
  • All required assessments available or user acknowledged gaps
  • Test results are complete (not partial or interrupted runs)
  • Test results match current codebase (not from outdated branch)

Knowledge Base Loading

  • risk-governance.md loaded successfully
  • probability-impact.md loaded successfully
  • test-quality.md loaded successfully
  • test-priorities.md loaded successfully
  • ci-burn-in.md loaded (if burn-in results available)

Process Steps

Step 1: Context Loading

  • Gate type identified (story/epic/release/hotfix)
  • Target ID extracted (story_id, epic_num, or release_version)
  • Decision thresholds loaded from workflow variables
  • Risk tolerance configuration loaded
  • Waiver policy loaded

Step 2: Evidence Parsing

Test Results:

  • Total test count extracted
  • Passed test count extracted
  • Failed test count extracted
  • Skipped test count extracted
  • Test duration extracted
  • P0 test pass rate calculated
  • P1 test pass rate calculated
  • Overall test pass rate calculated

Quality Assessments:

  • P0/P1/P2/P3 scenarios extracted from test-design.md (if available)
  • Risk scores extracted from test-design.md (if available)
  • Coverage percentages extracted from traceability-matrix.md (available from Phase 1)
  • Coverage gaps extracted from traceability-matrix.md (available from Phase 1)
  • NFR status extracted from nfr-assessment.md (if available)
  • Security issues count extracted from nfr-assessment.md (if available)

Code Coverage:

  • Line coverage percentage extracted (if available)
  • Branch coverage percentage extracted (if available)
  • Function coverage percentage extracted (if available)
  • Critical path coverage validated (if available)

Burn-in Results:

  • Burn-in iterations count extracted (if available)
  • Flaky tests count extracted (if available)
  • Stability score calculated (if available)

Step 3: Decision Rules Application

P0 Criteria Evaluation:

  • P0 test pass rate evaluated (must be 100%)
  • P0 acceptance criteria coverage evaluated (must be 100%)
  • Security issues count evaluated (must be 0)
  • Critical NFR failures evaluated (must be 0)
  • Flaky tests evaluated (must be 0 if burn-in enabled)
  • P0 decision recorded: PASS or FAIL

P1 Criteria Evaluation:

  • P1 test pass rate evaluated (threshold: min_p1_pass_rate)
  • P1 acceptance criteria coverage evaluated (threshold: 95%)
  • Overall test pass rate evaluated (threshold: min_overall_pass_rate)
  • Code coverage evaluated (threshold: min_coverage)
  • P1 decision recorded: PASS or CONCERNS

P2/P3 Criteria Evaluation:

  • P2 failures tracked (informational, don't block if allow_p2_failures: true)
  • P3 failures tracked (informational, don't block if allow_p3_failures: true)
  • Residual risks documented

Final Decision:

  • Decision determined: PASS / CONCERNS / FAIL / WAIVED
  • Decision rationale documented
  • Decision is deterministic (follows rules, not arbitrary)

Step 4: Documentation

Gate Decision Document Created:

  • Story/epic/release info section complete (ID, title, description, links)
  • Decision clearly stated (PASS / CONCERNS / FAIL / WAIVED)
  • Decision date recorded
  • Evaluator recorded (user or agent name)

Evidence Summary Documented:

  • Test results summary complete (total, passed, failed, pass rates)
  • Coverage summary complete (P0/P1 criteria, code coverage)
  • NFR validation summary complete (security, performance, reliability, maintainability)
  • Flakiness summary complete (burn-in iterations, flaky test count)

Rationale Documented:

  • Decision rationale clearly explained
  • Key evidence highlighted
  • Assumptions and caveats noted (if any)

Residual Risks Documented (if CONCERNS or WAIVED):

  • Unresolved P1/P2 issues listed
  • Probability × impact estimated for each risk
  • Mitigations or workarounds described

Waivers Documented (if WAIVED):

  • Waiver reason documented (business justification)
  • Waiver approver documented (name, role)
  • Waiver expiry date documented
  • Remediation plan documented (fix in next release, due date)
  • Monitoring plan documented

Critical Issues Documented (if FAIL or CONCERNS):

  • Top 5-10 critical issues listed
  • Priority assigned to each issue (P0/P1/P2)
  • Owner assigned to each issue
  • Due date assigned to each issue

Recommendations Documented:

  • Next steps clearly stated for decision type
  • Deployment recommendation provided
  • Monitoring recommendations provided (if applicable)
  • Remediation recommendations provided (if applicable)

Step 5: Status Updates and Notifications

Status File Updated:

  • Gate decision appended to bmm-workflow-status.md (if append_to_history: true)
  • Format correct: [DATE] Gate Decision: DECISION - Target {ID} - {rationale}
  • Status file committed or staged for commit

Gate YAML Created:

  • Gate YAML snippet generated with decision and criteria
  • Evidence references included in YAML
  • Next steps included in YAML
  • YAML file saved to output folder

Stakeholder Notification Generated:

  • Notification subject line created
  • Notification body created with summary
  • Recipients identified (PM, SM, DEV lead, stakeholders)
  • Notification ready for delivery (if notify_stakeholders: true)

Outputs Saved:

  • Gate decision document saved to {output_file}
  • Gate YAML saved to {output_folder}/gate-decision-{target}.yaml
  • All outputs are valid and readable

Phase 2 Output Validation

Gate Decision Document

Completeness:

  • All required sections present (info, decision, evidence, rationale, next steps)
  • No placeholder text or TODOs left in document
  • All evidence references are accurate and complete
  • All links to artifacts are valid

Accuracy:

  • Decision matches applied criteria rules
  • Test results match CI/CD pipeline output
  • Coverage percentages match reports
  • NFR status matches assessment document
  • No contradictions or inconsistencies

Clarity:

  • Decision rationale is clear and unambiguous
  • Technical jargon is explained or avoided
  • Stakeholders can understand next steps
  • Recommendations are actionable

Gate YAML

Format:

  • YAML is valid (no syntax errors)
  • All required fields present (target, decision, date, evaluator, criteria, evidence)
  • Field values are correct data types (numbers, strings, dates)

Content:

  • Criteria values match decision document
  • Evidence references are accurate
  • Next steps align with decision type

Phase 2 Quality Checks

Decision Integrity

  • Decision is deterministic (follows rules, not arbitrary)
  • P0 failures result in FAIL decision (unless waived)
  • Security issues result in FAIL decision (unless waived - but should never be waived)
  • Waivers have business justification and approver (if WAIVED)
  • Residual risks are documented (if CONCERNS or WAIVED)

Evidence-Based

  • Decision is based on actual test results (not guesses)
  • All claims are supported by evidence
  • No assumptions without documentation
  • Evidence sources are cited (CI run IDs, report URLs)

Transparency

  • Decision rationale is transparent and auditable
  • Criteria evaluation is documented step-by-step
  • Any deviations from standard process are explained
  • Waiver justifications are clear (if applicable)

Consistency

  • Decision aligns with risk-governance knowledge fragment
  • Priority framework (P0/P1/P2/P3) applied consistently
  • Terminology consistent with test-quality knowledge fragment
  • Decision matrix followed correctly

Phase 2 Integration Points

BMad Workflow Status

  • Gate decision added to bmm-workflow-status.md
  • Format matches existing gate history entries
  • Timestamp is accurate
  • Decision summary is concise (<80 chars)

CI/CD Pipeline

  • Gate YAML is CI/CD-compatible
  • YAML can be parsed by pipeline automation
  • Decision can be used to block/allow deployments
  • Evidence references are accessible to pipeline

Stakeholders

  • Notification message is clear and actionable
  • Decision is explained in non-technical terms
  • Next steps are specific and time-bound
  • Recipients are appropriate for decision type

Phase 2 Compliance and Audit

Audit Trail

  • Decision date and time recorded
  • Evaluator identified (user or agent)
  • All evidence sources cited
  • Decision criteria documented
  • Rationale clearly explained

Traceability

  • Gate decision traceable to story/epic/release
  • Evidence traceable to specific test runs
  • Assessments traceable to workflows that created them
  • Waiver traceable to approver (if applicable)

Compliance

  • Security requirements validated (no unresolved vulnerabilities)
  • Quality standards met or waived with justification
  • Regulatory requirements addressed (if applicable)
  • Documentation sufficient for external audit

Phase 2 Edge Cases and Exceptions

Missing Evidence

  • If test-design.md missing, decision still possible with test results + trace
  • If traceability-matrix.md missing, decision still possible with test results (but Phase 1 should provide it)
  • If nfr-assessment.md missing, NFR validation marked as NOT ASSESSED
  • If code coverage missing, coverage criterion marked as NOT ASSESSED
  • User acknowledged gaps in evidence or provided alternative proof

Stale Evidence

  • Evidence freshness checked (if validate_evidence_freshness: true)
  • Warnings issued for assessments >7 days old
  • User acknowledged stale evidence or re-ran workflows
  • Decision document notes any stale evidence used

Conflicting Evidence

  • Conflicts between test results and assessments resolved
  • Most recent/authoritative source identified
  • Conflict resolution documented in decision rationale
  • User consulted if conflict cannot be resolved

Waiver Scenarios

  • Waiver only used for FAIL decision (not PASS or CONCERNS)
  • Waiver has business justification (not technical convenience)
  • Waiver has named approver with authority (VP/CTO/PO)
  • Waiver has expiry date (does NOT apply to future releases)
  • Waiver has remediation plan with concrete due date
  • Security vulnerabilities are NOT waived (enforced)

FINAL VALIDATION (Both Phases)

Non-Prescriptive Validation

  • Traceability format adapted to team needs (not rigid template)
  • Examples are minimal and focused on patterns
  • Teams can extend with custom classifications
  • Integration with external systems supported (JIRA, Azure DevOps)
  • Compliance requirements considered (if applicable)

Documentation and Communication

  • All documents are readable and well-formatted
  • Tables render correctly in markdown
  • Code blocks have proper syntax highlighting
  • Links are valid and accessible
  • Recommendations are clear and prioritized
  • Gate decision is prominent and unambiguous (Phase 2)

Final Validation

Phase 1 (Traceability):

  • All prerequisites met
  • All acceptance criteria mapped or gaps documented
  • P0 coverage is 100% OR documented as BLOCKER
  • Gap analysis is complete and prioritized
  • Test quality issues identified and flagged
  • Deliverables generated and saved

Phase 2 (Gate Decision):

  • All quality evidence gathered
  • Decision criteria applied correctly
  • Decision rationale documented
  • Gate YAML ready for CI/CD integration
  • Status file updated (if enabled)
  • Stakeholders notified (if enabled)

Workflow Complete:

  • Phase 1 completed successfully
  • Phase 2 completed successfully (if enabled)
  • All outputs validated and saved
  • Ready to proceed based on gate decision

Sign-Off

Phase 1 - Traceability Status:

  • ✅ PASS - All quality gates met, no critical gaps
  • ⚠️ WARN - P1 gaps exist, address before PR merge
  • ❌ FAIL - P0 gaps exist, BLOCKER for release

Phase 2 - Gate Decision Status (if enabled):

  • ✅ PASS - Deploy to production
  • ⚠️ CONCERNS - Deploy with monitoring
  • ❌ FAIL - Block deployment, fix issues
  • 🔓 WAIVED - Deploy with business approval and remediation plan

Next Actions:

  • If PASS (both phases): Proceed to deployment
  • If WARN/CONCERNS: Address gaps/issues, proceed with monitoring
  • If FAIL (either phase): Run *atdd for missing tests, fix issues, re-run *trace
  • If WAIVED: Deploy with approved waiver, schedule remediation

Notes

Record any issues, deviations, or important observations during workflow execution:

  • Phase 1 Issues: [Note any traceability mapping challenges, missing tests, quality concerns]
  • Phase 2 Issues: [Note any missing, stale, or conflicting evidence]
  • Decision Rationale: [Document any nuanced reasoning or edge cases]
  • Waiver Details: [Document waiver negotiations or approvals]
  • Follow-up Actions: [List any actions required after gate decision]