bmad初始化
This commit is contained in:
493
bmad/bmm/workflows/testarch/test-design/README.md
Normal file
493
bmad/bmm/workflows/testarch/test-design/README.md
Normal file
@@ -0,0 +1,493 @@
|
||||
# Test Design and Risk Assessment Workflow
|
||||
|
||||
Plans comprehensive test coverage strategy with risk assessment (probability × impact scoring), priority classification (P0-P3), and resource estimation. This workflow generates a test design document that identifies high-risk areas, maps requirements to appropriate test levels, and provides execution ordering for optimal feedback.
|
||||
|
||||
## Usage
|
||||
|
||||
```bash
|
||||
bmad tea *test-design
|
||||
```
|
||||
|
||||
The TEA agent runs this workflow when:
|
||||
|
||||
- Planning test coverage before development starts
|
||||
- Assessing risks for an epic or story
|
||||
- Prioritizing test scenarios by business impact
|
||||
- Estimating testing effort and resources
|
||||
|
||||
## Inputs
|
||||
|
||||
**Required Context Files:**
|
||||
|
||||
- **Story markdown**: Acceptance criteria and requirements
|
||||
- **PRD or epics.md**: High-level product context
|
||||
- **Architecture docs** (optional): Technical constraints and integration points
|
||||
|
||||
**Workflow Variables:**
|
||||
|
||||
- `epic_num`: Epic number for scoped design
|
||||
- `story_path`: Specific story for design (optional)
|
||||
- `design_level`: full/targeted/minimal (default: full)
|
||||
- `risk_threshold`: Score for high-priority flag (default: 6)
|
||||
- `risk_categories`: TECH,SEC,PERF,DATA,BUS,OPS (all enabled)
|
||||
- `priority_levels`: P0,P1,P2,P3 (all enabled)
|
||||
|
||||
## Outputs
|
||||
|
||||
**Primary Deliverable:**
|
||||
|
||||
**Test Design Document** (`test-design-epic-{N}.md`):
|
||||
|
||||
1. **Risk Assessment Matrix**
|
||||
- Risk ID, category, description
|
||||
- Probability (1-3) × Impact (1-3) = Score
|
||||
- Scores ≥6 flagged as high-priority
|
||||
- Mitigation plans with owners and timelines
|
||||
|
||||
2. **Coverage Matrix**
|
||||
- Requirement → Test Level (E2E/API/Component/Unit)
|
||||
- Priority assignment (P0-P3)
|
||||
- Risk linkage
|
||||
- Test count estimates
|
||||
|
||||
3. **Execution Order**
|
||||
- Smoke tests (P0 subset, <5 min)
|
||||
- P0 tests (critical paths, <10 min)
|
||||
- P1 tests (important features, <30 min)
|
||||
- P2/P3 tests (full regression, <60 min)
|
||||
|
||||
4. **Resource Estimates**
|
||||
- Hours per priority level
|
||||
- Total effort in days
|
||||
- Tooling and data prerequisites
|
||||
|
||||
5. **Quality Gate Criteria**
|
||||
- P0 pass rate: 100%
|
||||
- P1 pass rate: ≥95%
|
||||
- High-risk mitigations: 100%
|
||||
- Coverage target: ≥80%
|
||||
|
||||
## Key Features
|
||||
|
||||
### Risk Scoring Framework
|
||||
|
||||
**Probability × Impact = Risk Score**
|
||||
|
||||
**Probability** (1-3):
|
||||
|
||||
- 1 (Unlikely): <10% chance
|
||||
- 2 (Possible): 10-50% chance
|
||||
- 3 (Likely): >50% chance
|
||||
|
||||
**Impact** (1-3):
|
||||
|
||||
- 1 (Minor): Cosmetic, workaround exists
|
||||
- 2 (Degraded): Feature impaired, difficult workaround
|
||||
- 3 (Critical): System failure, no workaround
|
||||
|
||||
**Scores**:
|
||||
|
||||
- 1-2: Low risk (monitor)
|
||||
- 3-4: Medium risk (plan mitigation)
|
||||
- **6-9: High risk** (immediate mitigation required)
|
||||
|
||||
### Risk Categories (6 types)
|
||||
|
||||
**TECH** (Technical/Architecture):
|
||||
|
||||
- Architecture flaws, integration failures
|
||||
- Scalability issues, technical debt
|
||||
|
||||
**SEC** (Security):
|
||||
|
||||
- Missing access controls, auth bypass
|
||||
- Data exposure, injection vulnerabilities
|
||||
|
||||
**PERF** (Performance):
|
||||
|
||||
- SLA violations, response time degradation
|
||||
- Resource exhaustion, scalability limits
|
||||
|
||||
**DATA** (Data Integrity):
|
||||
|
||||
- Data loss/corruption, inconsistent state
|
||||
- Migration failures
|
||||
|
||||
**BUS** (Business Impact):
|
||||
|
||||
- UX degradation, business logic errors
|
||||
- Revenue impact, compliance violations
|
||||
|
||||
**OPS** (Operations):
|
||||
|
||||
- Deployment failures, configuration errors
|
||||
- Monitoring gaps, rollback issues
|
||||
|
||||
### Priority Classification (P0-P3)
|
||||
|
||||
**P0 (Critical)** - Run on every commit:
|
||||
|
||||
- Blocks core user journey
|
||||
- High-risk (score ≥6)
|
||||
- Revenue-impacting or security-critical
|
||||
|
||||
**P1 (High)** - Run on PR to main:
|
||||
|
||||
- Important user features
|
||||
- Medium-risk (score 3-4)
|
||||
- Common workflows
|
||||
|
||||
**P2 (Medium)** - Run nightly/weekly:
|
||||
|
||||
- Secondary features
|
||||
- Low-risk (score 1-2)
|
||||
- Edge cases
|
||||
|
||||
**P3 (Low)** - Run on-demand:
|
||||
|
||||
- Nice-to-have, exploratory
|
||||
- Performance benchmarks
|
||||
|
||||
### Test Level Selection
|
||||
|
||||
**E2E (End-to-End)**:
|
||||
|
||||
- Critical user journeys
|
||||
- Multi-system integration
|
||||
- Highest confidence, slowest
|
||||
|
||||
**API (Integration)**:
|
||||
|
||||
- Service contracts
|
||||
- Business logic validation
|
||||
- Fast feedback, stable
|
||||
|
||||
**Component**:
|
||||
|
||||
- UI component behavior
|
||||
- Visual regression
|
||||
- Fast, isolated
|
||||
|
||||
**Unit**:
|
||||
|
||||
- Business logic, edge cases
|
||||
- Error handling
|
||||
- Fastest, most granular
|
||||
|
||||
**Key principle**: Avoid duplicate coverage - don't test same behavior at multiple levels.
|
||||
|
||||
### Exploratory Mode (NEW - Phase 2.5)
|
||||
|
||||
**test-design** supports UI exploration for brownfield applications with missing documentation.
|
||||
|
||||
**Activation**: Automatic when requirements missing/incomplete for brownfield apps
|
||||
|
||||
- If config.tea_use_mcp_enhancements is true + MCP available → MCP-assisted exploration
|
||||
- Otherwise → Manual exploration with user documentation
|
||||
|
||||
**When to Use Exploratory Mode:**
|
||||
|
||||
- ✅ Brownfield projects with missing documentation
|
||||
- ✅ Legacy systems lacking requirements
|
||||
- ✅ Undocumented features needing test coverage
|
||||
- ✅ Unknown user journeys requiring discovery
|
||||
- ❌ NOT for greenfield projects with clear requirements
|
||||
|
||||
**Exploration Modes:**
|
||||
|
||||
1. **MCP-Assisted Exploration** (if Playwright MCP available):
|
||||
- Interactive browser exploration using MCP tools
|
||||
- `planner_setup_page` - Initialize browser
|
||||
- `browser_navigate` - Explore pages
|
||||
- `browser_click` - Interact with UI elements
|
||||
- `browser_hover` - Reveal hidden menus
|
||||
- `browser_snapshot` - Capture state at each step
|
||||
- `browser_screenshot` - Document visually
|
||||
- `browser_console_messages` - Find JavaScript errors
|
||||
- `browser_network_requests` - Identify API endpoints
|
||||
|
||||
2. **Manual Exploration** (fallback without MCP):
|
||||
- User explores application manually
|
||||
- Documents findings in markdown:
|
||||
- Pages/features discovered
|
||||
- User journeys identified
|
||||
- API endpoints observed (DevTools Network)
|
||||
- JavaScript errors noted (DevTools Console)
|
||||
- Critical workflows mapped
|
||||
- Provides exploration findings to workflow
|
||||
|
||||
**Exploration Workflow:**
|
||||
|
||||
```
|
||||
1. Enable exploratory_mode and set exploration_url
|
||||
2. IF MCP available:
|
||||
- Use planner_setup_page to init browser
|
||||
- Explore UI with browser_* tools
|
||||
- Capture snapshots and screenshots
|
||||
- Monitor console and network
|
||||
- Document discoveries
|
||||
3. IF MCP unavailable:
|
||||
- Notify user to explore manually
|
||||
- Wait for exploration findings
|
||||
4. Convert discoveries to testable requirements
|
||||
5. Continue with standard risk assessment (Step 2)
|
||||
```
|
||||
|
||||
**Example Output from Exploratory Mode:**
|
||||
|
||||
```markdown
|
||||
## Exploration Findings - Legacy Admin Panel
|
||||
|
||||
**Exploration URL**: https://admin.example.com
|
||||
**Mode**: MCP-Assisted
|
||||
|
||||
### Discovered Features:
|
||||
|
||||
1. User Management (/admin/users)
|
||||
- List users (table with 10 columns)
|
||||
- Edit user (modal form)
|
||||
- Delete user (confirmation dialog)
|
||||
- Export to CSV (download button)
|
||||
|
||||
2. Reporting Dashboard (/admin/reports)
|
||||
- Date range picker
|
||||
- Filter by department
|
||||
- Generate PDF report
|
||||
- Email report to stakeholders
|
||||
|
||||
3. API Endpoints Discovered:
|
||||
- GET /api/admin/users
|
||||
- PUT /api/admin/users/:id
|
||||
- DELETE /api/admin/users/:id
|
||||
- POST /api/reports/generate
|
||||
|
||||
### User Journeys Mapped:
|
||||
|
||||
1. Admin deletes inactive user
|
||||
- Navigate to /admin/users
|
||||
- Click delete icon
|
||||
- Confirm in modal
|
||||
- User removed from table
|
||||
|
||||
2. Admin generates monthly report
|
||||
- Navigate to /admin/reports
|
||||
- Select date range (last month)
|
||||
- Click generate
|
||||
- Download PDF
|
||||
|
||||
### Risks Identified (from exploration):
|
||||
|
||||
- R-001 (SEC): No RBAC check observed (any admin can delete any user)
|
||||
- R-002 (DATA): No confirmation on bulk delete
|
||||
- R-003 (PERF): User table loads slowly (5s for 1000 rows)
|
||||
|
||||
**Next**: Proceed to risk assessment with discovered requirements
|
||||
```
|
||||
|
||||
**Graceful Degradation:**
|
||||
|
||||
- Exploratory mode is OPTIONAL (default: disabled)
|
||||
- Works without Playwright MCP (manual fallback)
|
||||
- If exploration fails, can disable mode and provide requirements documentation
|
||||
- Seamlessly transitions to standard risk assessment workflow
|
||||
|
||||
### Knowledge Base Integration
|
||||
|
||||
Automatically consults TEA knowledge base:
|
||||
|
||||
- `risk-governance.md` - Risk classification framework
|
||||
- `probability-impact.md` - Risk scoring methodology
|
||||
- `test-levels-framework.md` - Test level selection
|
||||
- `test-priorities-matrix.md` - P0-P3 prioritization
|
||||
|
||||
## Integration with Other Workflows
|
||||
|
||||
**Before test-design:**
|
||||
|
||||
- **prd** (Phase 2): Creates PRD and epics
|
||||
- **architecture** (Phase 3): Defines technical approach
|
||||
- **tech-spec** (Phase 3): Implementation details
|
||||
|
||||
**After test-design:**
|
||||
|
||||
- **atdd**: Generate failing tests for P0 scenarios
|
||||
- **automate**: Expand coverage for P1/P2 scenarios
|
||||
- **trace (Phase 2)**: Use quality gate criteria for release decisions
|
||||
|
||||
**Coordinates with:**
|
||||
|
||||
- **framework**: Test infrastructure must exist
|
||||
- **ci**: Execution order maps to CI stages
|
||||
|
||||
**Updates:**
|
||||
|
||||
- `bmm-workflow-status.md`: Adds test design to Quality & Testing Progress
|
||||
|
||||
## Important Notes
|
||||
|
||||
### Evidence-Based Assessment
|
||||
|
||||
**Critical principle**: Base risk assessment on **evidence**, not speculation.
|
||||
|
||||
**Evidence sources:**
|
||||
|
||||
- PRD and user research
|
||||
- Architecture documentation
|
||||
- Historical bug data
|
||||
- User feedback
|
||||
- Security audit results
|
||||
|
||||
**When uncertain**: Document assumptions, request user clarification.
|
||||
|
||||
**Avoid**:
|
||||
|
||||
- Guessing business impact
|
||||
- Assuming user behavior
|
||||
- Inventing requirements
|
||||
|
||||
### Resource Estimation Formula
|
||||
|
||||
```
|
||||
P0: 2 hours per test (setup + complex scenarios)
|
||||
P1: 1 hour per test (standard coverage)
|
||||
P2: 0.5 hours per test (simple scenarios)
|
||||
P3: 0.25 hours per test (exploratory)
|
||||
|
||||
Total Days = Total Hours / 8
|
||||
```
|
||||
|
||||
Example:
|
||||
|
||||
- 15 P0 × 2h = 30h
|
||||
- 25 P1 × 1h = 25h
|
||||
- 40 P2 × 0.5h = 20h
|
||||
- **Total: 75 hours (~10 days)**
|
||||
|
||||
### Execution Order Strategy
|
||||
|
||||
**Smoke tests** (subset of P0, <5 min):
|
||||
|
||||
- Login successful
|
||||
- Dashboard loads
|
||||
- Core API responds
|
||||
|
||||
**Purpose**: Fast feedback, catch build-breaking issues immediately.
|
||||
|
||||
**P0 tests** (critical paths, <10 min):
|
||||
|
||||
- All scenarios blocking user journeys
|
||||
- Security-critical flows
|
||||
|
||||
**P1 tests** (important features, <30 min):
|
||||
|
||||
- Common workflows
|
||||
- Medium-risk areas
|
||||
|
||||
**P2/P3 tests** (full regression, <60 min):
|
||||
|
||||
- Edge cases
|
||||
- Performance benchmarks
|
||||
|
||||
### Quality Gate Criteria
|
||||
|
||||
**Pass/Fail thresholds:**
|
||||
|
||||
- P0: 100% pass (no exceptions)
|
||||
- P1: ≥95% pass (2-3 failures acceptable with waivers)
|
||||
- P2/P3: ≥90% pass (informational)
|
||||
- High-risk items: All mitigated or have approved waivers
|
||||
|
||||
**Coverage targets:**
|
||||
|
||||
- Critical paths: ≥80%
|
||||
- Security scenarios: 100%
|
||||
- Business logic: ≥70%
|
||||
|
||||
## Validation Checklist
|
||||
|
||||
After workflow completion:
|
||||
|
||||
- [ ] Risk assessment complete (all categories)
|
||||
- [ ] Risks scored (probability × impact)
|
||||
- [ ] High-priority risks (≥6) flagged
|
||||
- [ ] Coverage matrix maps requirements to test levels
|
||||
- [ ] Priorities assigned (P0-P3)
|
||||
- [ ] Execution order defined
|
||||
- [ ] Resource estimates provided
|
||||
- [ ] Quality gate criteria defined
|
||||
- [ ] Output file created
|
||||
|
||||
Refer to `checklist.md` for comprehensive validation.
|
||||
|
||||
## Example Execution
|
||||
|
||||
**Scenario: E-commerce checkout epic**
|
||||
|
||||
```bash
|
||||
bmad tea *test-design
|
||||
# Epic 3: Checkout flow redesign
|
||||
|
||||
# Risk Assessment identifies:
|
||||
- R-001 (SEC): Payment bypass, P=2 × I=3 = 6 (HIGH)
|
||||
- R-002 (PERF): Cart load time, P=3 × I=2 = 6 (HIGH)
|
||||
- R-003 (BUS): Order confirmation email, P=2 × I=2 = 4 (MEDIUM)
|
||||
|
||||
# Coverage Plan:
|
||||
P0 scenarios: 12 tests (payment security, order creation)
|
||||
P1 scenarios: 18 tests (cart management, promo codes)
|
||||
P2 scenarios: 25 tests (edge cases, error handling)
|
||||
|
||||
Total effort: 65 hours (~8 days)
|
||||
|
||||
# Test Levels:
|
||||
- E2E: 8 tests (critical checkout path)
|
||||
- API: 30 tests (business logic, payment processing)
|
||||
- Unit: 17 tests (calculations, validations)
|
||||
|
||||
# Execution Order:
|
||||
1. Smoke: Payment successful, order created (2 min)
|
||||
2. P0: All payment & security flows (8 min)
|
||||
3. P1: Cart & promo codes (20 min)
|
||||
4. P2: Edge cases (40 min)
|
||||
|
||||
# Quality Gates:
|
||||
- P0 pass rate: 100%
|
||||
- P1 pass rate: ≥95%
|
||||
- R-001 mitigated: Add payment validation layer
|
||||
- R-002 mitigated: Implement cart caching
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**Issue: "Unable to score risks - missing context"**
|
||||
|
||||
- **Cause**: Insufficient documentation
|
||||
- **Solution**: Request PRD, architecture docs, or user clarification
|
||||
|
||||
**Issue: "All tests marked as P0"**
|
||||
|
||||
- **Cause**: Over-prioritization
|
||||
- **Solution**: Apply strict P0 criteria (blocks core journey + high risk + no workaround)
|
||||
|
||||
**Issue: "Duplicate coverage at multiple test levels"**
|
||||
|
||||
- **Cause**: Not following test pyramid
|
||||
- **Solution**: Use E2E for critical paths only, API for logic, unit for edge cases
|
||||
|
||||
**Issue: "Resource estimates too high"**
|
||||
|
||||
- **Cause**: Complex test setup or insufficient automation
|
||||
- **Solution**: Invest in fixtures/factories upfront, reduce per-test setup time
|
||||
|
||||
## Related Workflows
|
||||
|
||||
- **atdd**: Generate failing tests → [atdd/README.md](../atdd/README.md)
|
||||
- **automate**: Expand regression coverage → [automate/README.md](../automate/README.md)
|
||||
- **trace**: Traceability and quality gate decisions → [trace/README.md](../trace/README.md)
|
||||
- **framework**: Test infrastructure → [framework/README.md](../framework/README.md)
|
||||
|
||||
## Version History
|
||||
|
||||
- **v4.0 (BMad v6)**: Pure markdown instructions, risk scoring framework, template-based output
|
||||
- **v3.x**: XML format instructions
|
||||
- **v2.x**: Legacy task-based approach
|
||||
234
bmad/bmm/workflows/testarch/test-design/checklist.md
Normal file
234
bmad/bmm/workflows/testarch/test-design/checklist.md
Normal file
@@ -0,0 +1,234 @@
|
||||
# Test Design and Risk Assessment - Validation Checklist
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- [ ] Story markdown with clear acceptance criteria exists
|
||||
- [ ] PRD or epic documentation available
|
||||
- [ ] Architecture documents available (optional)
|
||||
- [ ] Requirements are testable and unambiguous
|
||||
|
||||
## Process Steps
|
||||
|
||||
### Step 1: Context Loading
|
||||
|
||||
- [ ] PRD.md read and requirements extracted
|
||||
- [ ] Epics.md or specific epic documentation loaded
|
||||
- [ ] Story markdown with acceptance criteria analyzed
|
||||
- [ ] Architecture documents reviewed (if available)
|
||||
- [ ] Existing test coverage analyzed
|
||||
- [ ] Knowledge base fragments loaded (risk-governance, probability-impact, test-levels, test-priorities)
|
||||
|
||||
### Step 2: Risk Assessment
|
||||
|
||||
- [ ] Genuine risks identified (not just features)
|
||||
- [ ] Risks classified by category (TECH/SEC/PERF/DATA/BUS/OPS)
|
||||
- [ ] Probability scored (1-3 for each risk)
|
||||
- [ ] Impact scored (1-3 for each risk)
|
||||
- [ ] Risk scores calculated (probability × impact)
|
||||
- [ ] High-priority risks (score ≥6) flagged
|
||||
- [ ] Mitigation plans defined for high-priority risks
|
||||
- [ ] Owners assigned for each mitigation
|
||||
- [ ] Timelines set for mitigations
|
||||
- [ ] Residual risk documented
|
||||
|
||||
### Step 3: Coverage Design
|
||||
|
||||
- [ ] Acceptance criteria broken into atomic scenarios
|
||||
- [ ] Test levels selected (E2E/API/Component/Unit)
|
||||
- [ ] No duplicate coverage across levels
|
||||
- [ ] Priority levels assigned (P0/P1/P2/P3)
|
||||
- [ ] P0 scenarios meet strict criteria (blocks core + high risk + no workaround)
|
||||
- [ ] Data prerequisites identified
|
||||
- [ ] Tooling requirements documented
|
||||
- [ ] Execution order defined (smoke → P0 → P1 → P2/P3)
|
||||
|
||||
### Step 4: Deliverables Generation
|
||||
|
||||
- [ ] Risk assessment matrix created
|
||||
- [ ] Coverage matrix created
|
||||
- [ ] Execution order documented
|
||||
- [ ] Resource estimates calculated
|
||||
- [ ] Quality gate criteria defined
|
||||
- [ ] Output file written to correct location
|
||||
- [ ] Output file uses template structure
|
||||
|
||||
## Output Validation
|
||||
|
||||
### Risk Assessment Matrix
|
||||
|
||||
- [ ] All risks have unique IDs (R-001, R-002, etc.)
|
||||
- [ ] Each risk has category assigned
|
||||
- [ ] Probability values are 1, 2, or 3
|
||||
- [ ] Impact values are 1, 2, or 3
|
||||
- [ ] Scores calculated correctly (P × I)
|
||||
- [ ] High-priority risks (≥6) clearly marked
|
||||
- [ ] Mitigation strategies specific and actionable
|
||||
|
||||
### Coverage Matrix
|
||||
|
||||
- [ ] All requirements mapped to test levels
|
||||
- [ ] Priorities assigned to all scenarios
|
||||
- [ ] Risk linkage documented
|
||||
- [ ] Test counts realistic
|
||||
- [ ] Owners assigned where applicable
|
||||
- [ ] No duplicate coverage (same behavior at multiple levels)
|
||||
|
||||
### Execution Order
|
||||
|
||||
- [ ] Smoke tests defined (<5 min target)
|
||||
- [ ] P0 tests listed (<10 min target)
|
||||
- [ ] P1 tests listed (<30 min target)
|
||||
- [ ] P2/P3 tests listed (<60 min target)
|
||||
- [ ] Order optimizes for fast feedback
|
||||
|
||||
### Resource Estimates
|
||||
|
||||
- [ ] P0 hours calculated (count × 2 hours)
|
||||
- [ ] P1 hours calculated (count × 1 hour)
|
||||
- [ ] P2 hours calculated (count × 0.5 hours)
|
||||
- [ ] P3 hours calculated (count × 0.25 hours)
|
||||
- [ ] Total hours summed
|
||||
- [ ] Days estimate provided (hours / 8)
|
||||
- [ ] Estimates include setup time
|
||||
|
||||
### Quality Gate Criteria
|
||||
|
||||
- [ ] P0 pass rate threshold defined (should be 100%)
|
||||
- [ ] P1 pass rate threshold defined (typically ≥95%)
|
||||
- [ ] High-risk mitigation completion required
|
||||
- [ ] Coverage targets specified (≥80% recommended)
|
||||
|
||||
## Quality Checks
|
||||
|
||||
### Evidence-Based Assessment
|
||||
|
||||
- [ ] Risk assessment based on documented evidence
|
||||
- [ ] No speculation on business impact
|
||||
- [ ] Assumptions clearly documented
|
||||
- [ ] Clarifications requested where needed
|
||||
- [ ] Historical data referenced where available
|
||||
|
||||
### Risk Classification Accuracy
|
||||
|
||||
- [ ] TECH risks are architecture/integration issues
|
||||
- [ ] SEC risks are security vulnerabilities
|
||||
- [ ] PERF risks are performance/scalability concerns
|
||||
- [ ] DATA risks are data integrity issues
|
||||
- [ ] BUS risks are business/revenue impacts
|
||||
- [ ] OPS risks are deployment/operational issues
|
||||
|
||||
### Priority Assignment Accuracy
|
||||
|
||||
- [ ] P0: Truly blocks core functionality
|
||||
- [ ] P0: High-risk (score ≥6)
|
||||
- [ ] P0: No workaround exists
|
||||
- [ ] P1: Important but not blocking
|
||||
- [ ] P2/P3: Nice-to-have or edge cases
|
||||
|
||||
### Test Level Selection
|
||||
|
||||
- [ ] E2E used only for critical paths
|
||||
- [ ] API tests cover complex business logic
|
||||
- [ ] Component tests for UI interactions
|
||||
- [ ] Unit tests for edge cases and algorithms
|
||||
- [ ] No redundant coverage
|
||||
|
||||
## Integration Points
|
||||
|
||||
### Knowledge Base Integration
|
||||
|
||||
- [ ] risk-governance.md consulted
|
||||
- [ ] probability-impact.md applied
|
||||
- [ ] test-levels-framework.md referenced
|
||||
- [ ] test-priorities-matrix.md used
|
||||
- [ ] Additional fragments loaded as needed
|
||||
|
||||
### Status File Integration
|
||||
|
||||
- [ ] bmm-workflow-status.md exists
|
||||
- [ ] Test design logged in Quality & Testing Progress
|
||||
- [ ] Epic number and scope documented
|
||||
- [ ] Completion timestamp recorded
|
||||
|
||||
### Workflow Dependencies
|
||||
|
||||
- [ ] Can proceed to `atdd` workflow with P0 scenarios
|
||||
- [ ] Can proceed to `automate` workflow with full coverage plan
|
||||
- [ ] Risk assessment informs `gate` workflow criteria
|
||||
- [ ] Integrates with `ci` workflow execution order
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
**All must be true:**
|
||||
|
||||
- [ ] All prerequisites met
|
||||
- [ ] All process steps completed
|
||||
- [ ] All output validations passed
|
||||
- [ ] All quality checks passed
|
||||
- [ ] All integration points verified
|
||||
- [ ] Output file complete and well-formatted
|
||||
- [ ] Team review scheduled (if required)
|
||||
|
||||
## Post-Workflow Actions
|
||||
|
||||
**User must complete:**
|
||||
|
||||
1. [ ] Review risk assessment with team
|
||||
2. [ ] Prioritize mitigation for high-priority risks (score ≥6)
|
||||
3. [ ] Allocate resources per estimates
|
||||
4. [ ] Run `atdd` workflow to generate P0 tests
|
||||
5. [ ] Set up test data factories and fixtures
|
||||
6. [ ] Schedule team review of test design document
|
||||
|
||||
**Recommended next workflows:**
|
||||
|
||||
1. [ ] Run `atdd` workflow for P0 test generation
|
||||
2. [ ] Run `framework` workflow if not already done
|
||||
3. [ ] Run `ci` workflow to configure pipeline stages
|
||||
|
||||
## Rollback Procedure
|
||||
|
||||
If workflow fails:
|
||||
|
||||
1. [ ] Delete output file
|
||||
2. [ ] Review error logs
|
||||
3. [ ] Fix missing context (PRD, architecture docs)
|
||||
4. [ ] Clarify ambiguous requirements
|
||||
5. [ ] Retry workflow
|
||||
|
||||
## Notes
|
||||
|
||||
### Common Issues
|
||||
|
||||
**Issue**: Too many P0 tests
|
||||
|
||||
- **Solution**: Apply strict P0 criteria - must block core AND high risk AND no workaround
|
||||
|
||||
**Issue**: Risk scores all high
|
||||
|
||||
- **Solution**: Differentiate between high-impact (3) and degraded (2) impacts
|
||||
|
||||
**Issue**: Duplicate coverage across levels
|
||||
|
||||
- **Solution**: Use test pyramid - E2E for critical paths only
|
||||
|
||||
**Issue**: Resource estimates too high
|
||||
|
||||
- **Solution**: Invest in fixtures/factories to reduce per-test setup time
|
||||
|
||||
### Best Practices
|
||||
|
||||
- Base risk assessment on evidence, not assumptions
|
||||
- High-priority risks (≥6) require immediate mitigation
|
||||
- P0 tests should cover <10% of total scenarios
|
||||
- Avoid testing same behavior at multiple levels
|
||||
- Include smoke tests (P0 subset) for fast feedback
|
||||
|
||||
---
|
||||
|
||||
**Checklist Complete**: Sign off when all items validated.
|
||||
|
||||
**Completed by:** **\*\***\_\_\_**\*\***
|
||||
**Date:** **\*\***\_\_\_**\*\***
|
||||
**Epic:** **\*\***\_\_\_**\*\***
|
||||
**Notes:** \***\*\*\*\*\***\*\*\***\*\*\*\*\***\_\_\_\***\*\*\*\*\***\*\*\***\*\*\*\*\***
|
||||
621
bmad/bmm/workflows/testarch/test-design/instructions.md
Normal file
621
bmad/bmm/workflows/testarch/test-design/instructions.md
Normal file
@@ -0,0 +1,621 @@
|
||||
<!-- Powered by BMAD-CORE™ -->
|
||||
|
||||
# Test Design and Risk Assessment
|
||||
|
||||
**Workflow ID**: `bmad/bmm/testarch/test-design`
|
||||
**Version**: 4.0 (BMad v6)
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
Plans comprehensive test coverage strategy with risk assessment, priority classification, and execution ordering. This workflow generates a test design document that identifies high-risk areas, maps requirements to test levels, prioritizes scenarios (P0-P3), and provides resource estimates for the testing effort.
|
||||
|
||||
---
|
||||
|
||||
## Preflight Requirements
|
||||
|
||||
**Critical:** Verify these requirements before proceeding. If any fail, HALT and notify the user.
|
||||
|
||||
- ✅ Story markdown with acceptance criteria available
|
||||
- ✅ PRD or epic documentation exists for context
|
||||
- ✅ Architecture documents available (optional but recommended)
|
||||
- ✅ Requirements are clear and testable
|
||||
|
||||
---
|
||||
|
||||
## Step 1: Load Context and Requirements
|
||||
|
||||
### Actions
|
||||
|
||||
1. **Read Requirements Documentation**
|
||||
- Load PRD.md for high-level product requirements
|
||||
- Read epics.md or specific epic for feature scope
|
||||
- Read story markdown for detailed acceptance criteria
|
||||
- Identify all testable requirements
|
||||
|
||||
2. **Load Architecture Context**
|
||||
- Read architecture.md for system design
|
||||
- Read tech-spec for implementation details
|
||||
- Identify technical constraints and dependencies
|
||||
- Note integration points and external systems
|
||||
|
||||
3. **Analyze Existing Test Coverage**
|
||||
- Search for existing test files in `{test_dir}`
|
||||
- Identify coverage gaps
|
||||
- Note areas with insufficient testing
|
||||
- Check for flaky or outdated tests
|
||||
|
||||
4. **Load Knowledge Base Fragments**
|
||||
|
||||
**Critical:** Consult `{project-root}/bmad/bmm/testarch/tea-index.csv` to load:
|
||||
- `risk-governance.md` - Risk classification framework (6 categories: TECH, SEC, PERF, DATA, BUS, OPS), automated scoring, gate decision engine, owner tracking (625 lines, 4 examples)
|
||||
- `probability-impact.md` - Risk scoring methodology (probability × impact matrix, automated classification, dynamic re-assessment, gate integration, 604 lines, 4 examples)
|
||||
- `test-levels-framework.md` - Test level selection guidance (E2E vs API vs Component vs Unit with decision matrix, characteristics, when to use each, 467 lines, 4 examples)
|
||||
- `test-priorities-matrix.md` - P0-P3 prioritization criteria (automated priority calculation, risk-based mapping, tagging strategy, time budgets, 389 lines, 2 examples)
|
||||
|
||||
**Halt Condition:** If story data or acceptance criteria are missing, check if brownfield exploration is needed. If neither requirements NOR exploration possible, HALT with message: "Test design requires clear requirements, acceptance criteria, or brownfield app URL for exploration"
|
||||
|
||||
---
|
||||
|
||||
## Step 1.5: Mode Selection (NEW - Phase 2.5)
|
||||
|
||||
### Actions
|
||||
|
||||
1. **Detect Planning Mode**
|
||||
|
||||
Determine mode based on context:
|
||||
|
||||
**Requirements-Based Mode (DEFAULT)**:
|
||||
- Have clear story/PRD with acceptance criteria
|
||||
- Uses: Existing workflow (Steps 2-4)
|
||||
- Appropriate for: Documented features, greenfield projects
|
||||
|
||||
**Exploratory Mode (OPTIONAL - Brownfield)**:
|
||||
- Missing/incomplete requirements AND brownfield application exists
|
||||
- Uses: UI exploration to discover functionality
|
||||
- Appropriate for: Undocumented brownfield apps, legacy systems
|
||||
|
||||
2. **Requirements-Based Mode (DEFAULT - Skip to Step 2)**
|
||||
|
||||
If requirements are clear:
|
||||
- Continue with existing workflow (Step 2: Assess and Classify Risks)
|
||||
- Use loaded requirements from Step 1
|
||||
- Proceed with risk assessment based on documented requirements
|
||||
|
||||
3. **Exploratory Mode (OPTIONAL - Brownfield Apps)**
|
||||
|
||||
If exploring brownfield application:
|
||||
|
||||
**A. Check MCP Availability**
|
||||
|
||||
If config.tea_use_mcp_enhancements is true AND Playwright MCP tools available:
|
||||
- Use MCP-assisted exploration (Step 3.B)
|
||||
|
||||
If MCP unavailable OR config.tea_use_mcp_enhancements is false:
|
||||
- Use manual exploration fallback (Step 3.C)
|
||||
|
||||
**B. MCP-Assisted Exploration (If MCP Tools Available)**
|
||||
|
||||
Use Playwright MCP browser tools to explore UI:
|
||||
|
||||
**Setup:**
|
||||
|
||||
```
|
||||
1. Use planner_setup_page to initialize browser
|
||||
2. Navigate to {exploration_url}
|
||||
3. Capture initial state with browser_snapshot
|
||||
```
|
||||
|
||||
**Exploration Process:**
|
||||
|
||||
```
|
||||
4. Use browser_navigate to explore different pages
|
||||
5. Use browser_click to interact with buttons, links, forms
|
||||
6. Use browser_hover to reveal hidden menus/tooltips
|
||||
7. Capture browser_snapshot at each significant state
|
||||
8. Take browser_screenshot for documentation
|
||||
9. Monitor browser_console_messages for JavaScript errors
|
||||
10. Track browser_network_requests to identify API calls
|
||||
11. Map user flows and interactive elements
|
||||
12. Document discovered functionality
|
||||
```
|
||||
|
||||
**Discovery Documentation:**
|
||||
- Create list of discovered features (pages, workflows, forms)
|
||||
- Identify user journeys (navigation paths)
|
||||
- Map API endpoints (from network requests)
|
||||
- Note error states (from console messages)
|
||||
- Capture screenshots for visual reference
|
||||
|
||||
**Convert to Test Scenarios:**
|
||||
- Transform discoveries into testable requirements
|
||||
- Prioritize based on user flow criticality
|
||||
- Identify risks from discovered functionality
|
||||
- Continue with Step 2 (Assess and Classify Risks) using discovered requirements
|
||||
|
||||
**C. Manual Exploration Fallback (If MCP Unavailable)**
|
||||
|
||||
If Playwright MCP is not available:
|
||||
|
||||
**Notify User:**
|
||||
|
||||
```markdown
|
||||
Exploratory mode enabled but Playwright MCP unavailable.
|
||||
|
||||
**Manual exploration required:**
|
||||
|
||||
1. Open application at: {exploration_url}
|
||||
2. Explore all pages, workflows, and features
|
||||
3. Document findings in markdown:
|
||||
- List of pages/features discovered
|
||||
- User journeys identified
|
||||
- API endpoints observed (DevTools Network tab)
|
||||
- JavaScript errors noted (DevTools Console)
|
||||
- Critical workflows mapped
|
||||
|
||||
4. Provide exploration findings to continue workflow
|
||||
|
||||
**Alternative:** Disable exploratory_mode and provide requirements documentation
|
||||
```
|
||||
|
||||
Wait for user to provide exploration findings, then:
|
||||
- Parse user-provided discovery documentation
|
||||
- Convert to testable requirements
|
||||
- Continue with Step 2 (risk assessment)
|
||||
|
||||
4. **Proceed to Risk Assessment**
|
||||
|
||||
After mode selection (Requirements-Based OR Exploratory):
|
||||
- Continue to Step 2: Assess and Classify Risks
|
||||
- Use requirements from documentation (Requirements-Based) OR discoveries (Exploratory)
|
||||
|
||||
---
|
||||
|
||||
## Step 2: Assess and Classify Risks
|
||||
|
||||
### Actions
|
||||
|
||||
1. **Identify Genuine Risks**
|
||||
|
||||
Filter requirements to isolate actual risks (not just features):
|
||||
- Unresolved technical gaps
|
||||
- Security vulnerabilities
|
||||
- Performance bottlenecks
|
||||
- Data loss or corruption potential
|
||||
- Business impact failures
|
||||
- Operational deployment issues
|
||||
|
||||
2. **Classify Risks by Category**
|
||||
|
||||
Use these standard risk categories:
|
||||
|
||||
**TECH** (Technical/Architecture):
|
||||
- Architecture flaws
|
||||
- Integration failures
|
||||
- Scalability issues
|
||||
- Technical debt
|
||||
|
||||
**SEC** (Security):
|
||||
- Missing access controls
|
||||
- Authentication bypass
|
||||
- Data exposure
|
||||
- Injection vulnerabilities
|
||||
|
||||
**PERF** (Performance):
|
||||
- SLA violations
|
||||
- Response time degradation
|
||||
- Resource exhaustion
|
||||
- Scalability limits
|
||||
|
||||
**DATA** (Data Integrity):
|
||||
- Data loss
|
||||
- Data corruption
|
||||
- Inconsistent state
|
||||
- Migration failures
|
||||
|
||||
**BUS** (Business Impact):
|
||||
- User experience degradation
|
||||
- Business logic errors
|
||||
- Revenue impact
|
||||
- Compliance violations
|
||||
|
||||
**OPS** (Operations):
|
||||
- Deployment failures
|
||||
- Configuration errors
|
||||
- Monitoring gaps
|
||||
- Rollback issues
|
||||
|
||||
3. **Score Risk Probability**
|
||||
|
||||
Rate likelihood (1-3):
|
||||
- **1 (Unlikely)**: <10% chance, edge case
|
||||
- **2 (Possible)**: 10-50% chance, known scenario
|
||||
- **3 (Likely)**: >50% chance, common occurrence
|
||||
|
||||
4. **Score Risk Impact**
|
||||
|
||||
Rate severity (1-3):
|
||||
- **1 (Minor)**: Cosmetic, workaround exists, limited users
|
||||
- **2 (Degraded)**: Feature impaired, workaround difficult, affects many users
|
||||
- **3 (Critical)**: System failure, data loss, no workaround, blocks usage
|
||||
|
||||
5. **Calculate Risk Score**
|
||||
|
||||
```
|
||||
Risk Score = Probability × Impact
|
||||
|
||||
Scores:
|
||||
1-2: Low risk (monitor)
|
||||
3-4: Medium risk (plan mitigation)
|
||||
6-9: High risk (immediate mitigation required)
|
||||
```
|
||||
|
||||
6. **Highlight High-Priority Risks**
|
||||
|
||||
Flag all risks with score ≥6 for immediate attention.
|
||||
|
||||
7. **Request Clarification**
|
||||
|
||||
If evidence is missing or assumptions required:
|
||||
- Document assumptions clearly
|
||||
- Request user clarification
|
||||
- Do NOT speculate on business impact
|
||||
|
||||
8. **Plan Mitigations**
|
||||
|
||||
For each high-priority risk:
|
||||
- Define mitigation strategy
|
||||
- Assign owner (dev, QA, ops)
|
||||
- Set timeline
|
||||
- Update residual risk expectation
|
||||
|
||||
---
|
||||
|
||||
## Step 3: Design Test Coverage
|
||||
|
||||
### Actions
|
||||
|
||||
1. **Break Down Acceptance Criteria**
|
||||
|
||||
Convert each acceptance criterion into atomic test scenarios:
|
||||
- One scenario per testable behavior
|
||||
- Scenarios are independent
|
||||
- Scenarios are repeatable
|
||||
- Scenarios tie back to risk mitigations
|
||||
|
||||
2. **Select Appropriate Test Levels**
|
||||
|
||||
**Knowledge Base Reference**: `test-levels-framework.md`
|
||||
|
||||
Map requirements to optimal test levels (avoid duplication):
|
||||
|
||||
**E2E (End-to-End)**:
|
||||
- Critical user journeys
|
||||
- Multi-system integration
|
||||
- Production-like environment
|
||||
- Highest confidence, slowest execution
|
||||
|
||||
**API (Integration)**:
|
||||
- Service contracts
|
||||
- Business logic validation
|
||||
- Fast feedback
|
||||
- Good for complex scenarios
|
||||
|
||||
**Component**:
|
||||
- UI component behavior
|
||||
- Interaction testing
|
||||
- Visual regression
|
||||
- Fast, isolated
|
||||
|
||||
**Unit**:
|
||||
- Business logic
|
||||
- Edge cases
|
||||
- Error handling
|
||||
- Fastest, most granular
|
||||
|
||||
**Avoid duplicate coverage**: Don't test same behavior at multiple levels unless necessary.
|
||||
|
||||
3. **Assign Priority Levels**
|
||||
|
||||
**Knowledge Base Reference**: `test-priorities-matrix.md`
|
||||
|
||||
**P0 (Critical)**:
|
||||
- Blocks core user journey
|
||||
- High-risk areas (score ≥6)
|
||||
- Revenue-impacting
|
||||
- Security-critical
|
||||
- **Run on every commit**
|
||||
|
||||
**P1 (High)**:
|
||||
- Important user features
|
||||
- Medium-risk areas (score 3-4)
|
||||
- Common workflows
|
||||
- **Run on PR to main**
|
||||
|
||||
**P2 (Medium)**:
|
||||
- Secondary features
|
||||
- Low-risk areas (score 1-2)
|
||||
- Edge cases
|
||||
- **Run nightly or weekly**
|
||||
|
||||
**P3 (Low)**:
|
||||
- Nice-to-have
|
||||
- Exploratory
|
||||
- Performance benchmarks
|
||||
- **Run on-demand**
|
||||
|
||||
4. **Outline Data and Tooling Prerequisites**
|
||||
|
||||
For each test scenario, identify:
|
||||
- Test data requirements (factories, fixtures)
|
||||
- External services (mocks, stubs)
|
||||
- Environment setup
|
||||
- Tools and dependencies
|
||||
|
||||
5. **Define Execution Order**
|
||||
|
||||
Recommend test execution sequence:
|
||||
1. **Smoke tests** (P0 subset, <5 min)
|
||||
2. **P0 tests** (critical paths, <10 min)
|
||||
3. **P1 tests** (important features, <30 min)
|
||||
4. **P2/P3 tests** (full regression, <60 min)
|
||||
|
||||
---
|
||||
|
||||
## Step 4: Generate Deliverables
|
||||
|
||||
### Actions
|
||||
|
||||
1. **Create Risk Assessment Matrix**
|
||||
|
||||
Use template structure:
|
||||
|
||||
```markdown
|
||||
| Risk ID | Category | Description | Probability | Impact | Score | Mitigation |
|
||||
| ------- | -------- | ----------- | ----------- | ------ | ----- | --------------- |
|
||||
| R-001 | SEC | Auth bypass | 2 | 3 | 6 | Add authz check |
|
||||
```
|
||||
|
||||
2. **Create Coverage Matrix**
|
||||
|
||||
```markdown
|
||||
| Requirement | Test Level | Priority | Risk Link | Test Count | Owner |
|
||||
| ----------- | ---------- | -------- | --------- | ---------- | ----- |
|
||||
| Login flow | E2E | P0 | R-001 | 3 | QA |
|
||||
```
|
||||
|
||||
3. **Document Execution Order**
|
||||
|
||||
```markdown
|
||||
### Smoke Tests (<5 min)
|
||||
|
||||
- Login successful
|
||||
- Dashboard loads
|
||||
|
||||
### P0 Tests (<10 min)
|
||||
|
||||
- [Full P0 list]
|
||||
|
||||
### P1 Tests (<30 min)
|
||||
|
||||
- [Full P1 list]
|
||||
```
|
||||
|
||||
4. **Include Resource Estimates**
|
||||
|
||||
```markdown
|
||||
### Test Effort Estimates
|
||||
|
||||
- P0 scenarios: 15 tests × 2 hours = 30 hours
|
||||
- P1 scenarios: 25 tests × 1 hour = 25 hours
|
||||
- P2 scenarios: 40 tests × 0.5 hour = 20 hours
|
||||
- **Total:** 75 hours (~10 days)
|
||||
```
|
||||
|
||||
5. **Add Gate Criteria**
|
||||
|
||||
```markdown
|
||||
### Quality Gate Criteria
|
||||
|
||||
- All P0 tests pass (100%)
|
||||
- P1 tests pass rate ≥95%
|
||||
- No high-risk (score ≥6) items unmitigated
|
||||
- Test coverage ≥80% for critical paths
|
||||
```
|
||||
|
||||
6. **Write to Output File**
|
||||
|
||||
Save to `{output_folder}/test-design-epic-{epic_num}.md` using template structure.
|
||||
|
||||
---
|
||||
|
||||
## Important Notes
|
||||
|
||||
### Risk Category Definitions
|
||||
|
||||
**TECH** (Technical/Architecture):
|
||||
|
||||
- Architecture flaws or technical debt
|
||||
- Integration complexity
|
||||
- Scalability concerns
|
||||
|
||||
**SEC** (Security):
|
||||
|
||||
- Missing security controls
|
||||
- Authentication/authorization gaps
|
||||
- Data exposure risks
|
||||
|
||||
**PERF** (Performance):
|
||||
|
||||
- SLA risk or performance degradation
|
||||
- Resource constraints
|
||||
- Scalability bottlenecks
|
||||
|
||||
**DATA** (Data Integrity):
|
||||
|
||||
- Data loss or corruption potential
|
||||
- State consistency issues
|
||||
- Migration risks
|
||||
|
||||
**BUS** (Business Impact):
|
||||
|
||||
- User experience harm
|
||||
- Business logic errors
|
||||
- Revenue or compliance impact
|
||||
|
||||
**OPS** (Operations):
|
||||
|
||||
- Deployment or runtime failures
|
||||
- Configuration issues
|
||||
- Monitoring/observability gaps
|
||||
|
||||
### Risk Scoring Methodology
|
||||
|
||||
**Probability × Impact = Risk Score**
|
||||
|
||||
Examples:
|
||||
|
||||
- High likelihood (3) × Critical impact (3) = **Score 9** (highest priority)
|
||||
- Possible (2) × Critical (3) = **Score 6** (high priority threshold)
|
||||
- Unlikely (1) × Minor (1) = **Score 1** (low priority)
|
||||
|
||||
**Threshold**: Scores ≥6 require immediate mitigation.
|
||||
|
||||
### Test Level Selection Strategy
|
||||
|
||||
**Avoid duplication:**
|
||||
|
||||
- Don't test same behavior at E2E and API level
|
||||
- Use E2E for critical paths only
|
||||
- Use API tests for complex business logic
|
||||
- Use unit tests for edge cases
|
||||
|
||||
**Tradeoffs:**
|
||||
|
||||
- E2E: High confidence, slow execution, brittle
|
||||
- API: Good balance, fast, stable
|
||||
- Unit: Fastest feedback, narrow scope
|
||||
|
||||
### Priority Assignment Guidelines
|
||||
|
||||
**P0 criteria** (all must be true):
|
||||
|
||||
- Blocks core functionality
|
||||
- High-risk (score ≥6)
|
||||
- No workaround exists
|
||||
- Affects majority of users
|
||||
|
||||
**P1 criteria**:
|
||||
|
||||
- Important feature
|
||||
- Medium risk (score 3-5)
|
||||
- Workaround exists but difficult
|
||||
|
||||
**P2/P3**: Everything else, prioritized by value
|
||||
|
||||
### Knowledge Base Integration
|
||||
|
||||
**Core Fragments (Auto-loaded in Step 1):**
|
||||
|
||||
- `risk-governance.md` - Risk classification (6 categories), automated scoring, gate decision engine, coverage traceability, owner tracking (625 lines, 4 examples)
|
||||
- `probability-impact.md` - Probability × impact matrix, automated classification thresholds, dynamic re-assessment, gate integration (604 lines, 4 examples)
|
||||
- `test-levels-framework.md` - E2E vs API vs Component vs Unit decision framework with characteristics matrix (467 lines, 4 examples)
|
||||
- `test-priorities-matrix.md` - P0-P3 automated priority calculation, risk-based mapping, tagging strategy, time budgets (389 lines, 2 examples)
|
||||
|
||||
**Reference for Test Planning:**
|
||||
|
||||
- `selective-testing.md` - Execution strategy: tag-based, spec filters, diff-based selection, promotion rules (727 lines, 4 examples)
|
||||
- `fixture-architecture.md` - Data setup patterns: pure function → fixture → mergeTests, auto-cleanup (406 lines, 5 examples)
|
||||
|
||||
**Manual Reference (Optional):**
|
||||
|
||||
- Use `tea-index.csv` to find additional specialized fragments as needed
|
||||
|
||||
### Evidence-Based Assessment
|
||||
|
||||
**Critical principle:** Base risk assessment on evidence, not speculation.
|
||||
|
||||
**Evidence sources:**
|
||||
|
||||
- PRD and user research
|
||||
- Architecture documentation
|
||||
- Historical bug data
|
||||
- User feedback
|
||||
- Security audit results
|
||||
|
||||
**Avoid:**
|
||||
|
||||
- Guessing business impact
|
||||
- Assuming user behavior
|
||||
- Inventing requirements
|
||||
|
||||
**When uncertain:** Document assumptions and request clarification from user.
|
||||
|
||||
---
|
||||
|
||||
## Output Summary
|
||||
|
||||
After completing this workflow, provide a summary:
|
||||
|
||||
```markdown
|
||||
## Test Design Complete
|
||||
|
||||
**Epic**: {epic_num}
|
||||
**Scope**: {design_level}
|
||||
|
||||
**Risk Assessment**:
|
||||
|
||||
- Total risks identified: {count}
|
||||
- High-priority risks (≥6): {high_count}
|
||||
- Categories: {categories}
|
||||
|
||||
**Coverage Plan**:
|
||||
|
||||
- P0 scenarios: {p0_count} ({p0_hours} hours)
|
||||
- P1 scenarios: {p1_count} ({p1_hours} hours)
|
||||
- P2/P3 scenarios: {p2p3_count} ({p2p3_hours} hours)
|
||||
- **Total effort**: {total_hours} hours (~{total_days} days)
|
||||
|
||||
**Test Levels**:
|
||||
|
||||
- E2E: {e2e_count}
|
||||
- API: {api_count}
|
||||
- Component: {component_count}
|
||||
- Unit: {unit_count}
|
||||
|
||||
**Quality Gate Criteria**:
|
||||
|
||||
- P0 pass rate: 100%
|
||||
- P1 pass rate: ≥95%
|
||||
- High-risk mitigations: 100%
|
||||
- Coverage: ≥80%
|
||||
|
||||
**Output File**: {output_file}
|
||||
|
||||
**Next Steps**:
|
||||
|
||||
1. Review risk assessment with team
|
||||
2. Prioritize mitigation for high-risk items (score ≥6)
|
||||
3. Run `atdd` workflow to generate failing tests for P0 scenarios
|
||||
4. Allocate resources per effort estimates
|
||||
5. Set up test data factories and fixtures
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Validation
|
||||
|
||||
After completing all steps, verify:
|
||||
|
||||
- [ ] Risk assessment complete with all categories
|
||||
- [ ] All risks scored (probability × impact)
|
||||
- [ ] High-priority risks (≥6) flagged
|
||||
- [ ] Coverage matrix maps requirements to test levels
|
||||
- [ ] Priority levels assigned (P0-P3)
|
||||
- [ ] Execution order defined
|
||||
- [ ] Resource estimates provided
|
||||
- [ ] Quality gate criteria defined
|
||||
- [ ] Output file created and formatted correctly
|
||||
|
||||
Refer to `checklist.md` for comprehensive validation criteria.
|
||||
285
bmad/bmm/workflows/testarch/test-design/test-design-template.md
Normal file
285
bmad/bmm/workflows/testarch/test-design/test-design-template.md
Normal file
@@ -0,0 +1,285 @@
|
||||
# Test Design: Epic {epic_num} - {epic_title}
|
||||
|
||||
**Date:** {date}
|
||||
**Author:** {user_name}
|
||||
**Status:** Draft / Approved
|
||||
|
||||
---
|
||||
|
||||
## Executive Summary
|
||||
|
||||
**Scope:** {design_level} test design for Epic {epic_num}
|
||||
|
||||
**Risk Summary:**
|
||||
|
||||
- Total risks identified: {total_risks}
|
||||
- High-priority risks (≥6): {high_priority_count}
|
||||
- Critical categories: {top_categories}
|
||||
|
||||
**Coverage Summary:**
|
||||
|
||||
- P0 scenarios: {p0_count} ({p0_hours} hours)
|
||||
- P1 scenarios: {p1_count} ({p1_hours} hours)
|
||||
- P2/P3 scenarios: {p2p3_count} ({p2p3_hours} hours)
|
||||
- **Total effort**: {total_hours} hours (~{total_days} days)
|
||||
|
||||
---
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
### High-Priority Risks (Score ≥6)
|
||||
|
||||
| Risk ID | Category | Description | Probability | Impact | Score | Mitigation | Owner | Timeline |
|
||||
| ------- | -------- | ------------- | ----------- | ------ | ----- | ------------ | ------- | -------- |
|
||||
| R-001 | SEC | {description} | 2 | 3 | 6 | {mitigation} | {owner} | {date} |
|
||||
| R-002 | PERF | {description} | 3 | 2 | 6 | {mitigation} | {owner} | {date} |
|
||||
|
||||
### Medium-Priority Risks (Score 3-4)
|
||||
|
||||
| Risk ID | Category | Description | Probability | Impact | Score | Mitigation | Owner |
|
||||
| ------- | -------- | ------------- | ----------- | ------ | ----- | ------------ | ------- |
|
||||
| R-003 | TECH | {description} | 2 | 2 | 4 | {mitigation} | {owner} |
|
||||
| R-004 | DATA | {description} | 1 | 3 | 3 | {mitigation} | {owner} |
|
||||
|
||||
### Low-Priority Risks (Score 1-2)
|
||||
|
||||
| Risk ID | Category | Description | Probability | Impact | Score | Action |
|
||||
| ------- | -------- | ------------- | ----------- | ------ | ----- | ------- |
|
||||
| R-005 | OPS | {description} | 1 | 2 | 2 | Monitor |
|
||||
| R-006 | BUS | {description} | 1 | 1 | 1 | Monitor |
|
||||
|
||||
### Risk Category Legend
|
||||
|
||||
- **TECH**: Technical/Architecture (flaws, integration, scalability)
|
||||
- **SEC**: Security (access controls, auth, data exposure)
|
||||
- **PERF**: Performance (SLA violations, degradation, resource limits)
|
||||
- **DATA**: Data Integrity (loss, corruption, inconsistency)
|
||||
- **BUS**: Business Impact (UX harm, logic errors, revenue)
|
||||
- **OPS**: Operations (deployment, config, monitoring)
|
||||
|
||||
---
|
||||
|
||||
## Test Coverage Plan
|
||||
|
||||
### P0 (Critical) - Run on every commit
|
||||
|
||||
**Criteria**: Blocks core journey + High risk (≥6) + No workaround
|
||||
|
||||
| Requirement | Test Level | Risk Link | Test Count | Owner | Notes |
|
||||
| ------------- | ---------- | --------- | ---------- | ----- | ------- |
|
||||
| {requirement} | E2E | R-001 | 3 | QA | {notes} |
|
||||
| {requirement} | API | R-002 | 5 | QA | {notes} |
|
||||
|
||||
**Total P0**: {p0_count} tests, {p0_hours} hours
|
||||
|
||||
### P1 (High) - Run on PR to main
|
||||
|
||||
**Criteria**: Important features + Medium risk (3-4) + Common workflows
|
||||
|
||||
| Requirement | Test Level | Risk Link | Test Count | Owner | Notes |
|
||||
| ------------- | ---------- | --------- | ---------- | ----- | ------- |
|
||||
| {requirement} | API | R-003 | 4 | QA | {notes} |
|
||||
| {requirement} | Component | - | 6 | DEV | {notes} |
|
||||
|
||||
**Total P1**: {p1_count} tests, {p1_hours} hours
|
||||
|
||||
### P2 (Medium) - Run nightly/weekly
|
||||
|
||||
**Criteria**: Secondary features + Low risk (1-2) + Edge cases
|
||||
|
||||
| Requirement | Test Level | Risk Link | Test Count | Owner | Notes |
|
||||
| ------------- | ---------- | --------- | ---------- | ----- | ------- |
|
||||
| {requirement} | API | R-004 | 8 | QA | {notes} |
|
||||
| {requirement} | Unit | - | 15 | DEV | {notes} |
|
||||
|
||||
**Total P2**: {p2_count} tests, {p2_hours} hours
|
||||
|
||||
### P3 (Low) - Run on-demand
|
||||
|
||||
**Criteria**: Nice-to-have + Exploratory + Performance benchmarks
|
||||
|
||||
| Requirement | Test Level | Test Count | Owner | Notes |
|
||||
| ------------- | ---------- | ---------- | ----- | ------- |
|
||||
| {requirement} | E2E | 2 | QA | {notes} |
|
||||
| {requirement} | Unit | 8 | DEV | {notes} |
|
||||
|
||||
**Total P3**: {p3_count} tests, {p3_hours} hours
|
||||
|
||||
---
|
||||
|
||||
## Execution Order
|
||||
|
||||
### Smoke Tests (<5 min)
|
||||
|
||||
**Purpose**: Fast feedback, catch build-breaking issues
|
||||
|
||||
- [ ] {scenario} (30s)
|
||||
- [ ] {scenario} (45s)
|
||||
- [ ] {scenario} (1min)
|
||||
|
||||
**Total**: {smoke_count} scenarios
|
||||
|
||||
### P0 Tests (<10 min)
|
||||
|
||||
**Purpose**: Critical path validation
|
||||
|
||||
- [ ] {scenario} (E2E)
|
||||
- [ ] {scenario} (API)
|
||||
- [ ] {scenario} (API)
|
||||
|
||||
**Total**: {p0_count} scenarios
|
||||
|
||||
### P1 Tests (<30 min)
|
||||
|
||||
**Purpose**: Important feature coverage
|
||||
|
||||
- [ ] {scenario} (API)
|
||||
- [ ] {scenario} (Component)
|
||||
|
||||
**Total**: {p1_count} scenarios
|
||||
|
||||
### P2/P3 Tests (<60 min)
|
||||
|
||||
**Purpose**: Full regression coverage
|
||||
|
||||
- [ ] {scenario} (Unit)
|
||||
- [ ] {scenario} (API)
|
||||
|
||||
**Total**: {p2p3_count} scenarios
|
||||
|
||||
---
|
||||
|
||||
## Resource Estimates
|
||||
|
||||
### Test Development Effort
|
||||
|
||||
| Priority | Count | Hours/Test | Total Hours | Notes |
|
||||
| --------- | ----------------- | ---------- | ----------------- | ----------------------- |
|
||||
| P0 | {p0_count} | 2.0 | {p0_hours} | Complex setup, security |
|
||||
| P1 | {p1_count} | 1.0 | {p1_hours} | Standard coverage |
|
||||
| P2 | {p2_count} | 0.5 | {p2_hours} | Simple scenarios |
|
||||
| P3 | {p3_count} | 0.25 | {p3_hours} | Exploratory |
|
||||
| **Total** | **{total_count}** | **-** | **{total_hours}** | **~{total_days} days** |
|
||||
|
||||
### Prerequisites
|
||||
|
||||
**Test Data:**
|
||||
|
||||
- {factory_name} factory (faker-based, auto-cleanup)
|
||||
- {fixture_name} fixture (setup/teardown)
|
||||
|
||||
**Tooling:**
|
||||
|
||||
- {tool} for {purpose}
|
||||
- {tool} for {purpose}
|
||||
|
||||
**Environment:**
|
||||
|
||||
- {env_requirement}
|
||||
- {env_requirement}
|
||||
|
||||
---
|
||||
|
||||
## Quality Gate Criteria
|
||||
|
||||
### Pass/Fail Thresholds
|
||||
|
||||
- **P0 pass rate**: 100% (no exceptions)
|
||||
- **P1 pass rate**: ≥95% (waivers required for failures)
|
||||
- **P2/P3 pass rate**: ≥90% (informational)
|
||||
- **High-risk mitigations**: 100% complete or approved waivers
|
||||
|
||||
### Coverage Targets
|
||||
|
||||
- **Critical paths**: ≥80%
|
||||
- **Security scenarios**: 100%
|
||||
- **Business logic**: ≥70%
|
||||
- **Edge cases**: ≥50%
|
||||
|
||||
### Non-Negotiable Requirements
|
||||
|
||||
- [ ] All P0 tests pass
|
||||
- [ ] No high-risk (≥6) items unmitigated
|
||||
- [ ] Security tests (SEC category) pass 100%
|
||||
- [ ] Performance targets met (PERF category)
|
||||
|
||||
---
|
||||
|
||||
## Mitigation Plans
|
||||
|
||||
### R-001: {Risk Description} (Score: 6)
|
||||
|
||||
**Mitigation Strategy:** {detailed_mitigation}
|
||||
**Owner:** {owner}
|
||||
**Timeline:** {date}
|
||||
**Status:** Planned / In Progress / Complete
|
||||
**Verification:** {how_to_verify}
|
||||
|
||||
### R-002: {Risk Description} (Score: 6)
|
||||
|
||||
**Mitigation Strategy:** {detailed_mitigation}
|
||||
**Owner:** {owner}
|
||||
**Timeline:** {date}
|
||||
**Status:** Planned / In Progress / Complete
|
||||
**Verification:** {how_to_verify}
|
||||
|
||||
---
|
||||
|
||||
## Assumptions and Dependencies
|
||||
|
||||
### Assumptions
|
||||
|
||||
1. {assumption}
|
||||
2. {assumption}
|
||||
3. {assumption}
|
||||
|
||||
### Dependencies
|
||||
|
||||
1. {dependency} - Required by {date}
|
||||
2. {dependency} - Required by {date}
|
||||
|
||||
### Risks to Plan
|
||||
|
||||
- **Risk**: {risk_to_plan}
|
||||
- **Impact**: {impact}
|
||||
- **Contingency**: {contingency}
|
||||
|
||||
---
|
||||
|
||||
## Approval
|
||||
|
||||
**Test Design Approved By:**
|
||||
|
||||
- [ ] Product Manager: **\*\***\_\_\_**\*\*** Date: **\*\***\_\_\_**\*\***
|
||||
- [ ] Tech Lead: **\*\***\_\_\_**\*\*** Date: **\*\***\_\_\_**\*\***
|
||||
- [ ] QA Lead: **\*\***\_\_\_**\*\*** Date: **\*\***\_\_\_**\*\***
|
||||
|
||||
**Comments:**
|
||||
|
||||
---
|
||||
|
||||
---
|
||||
|
||||
---
|
||||
|
||||
## Appendix
|
||||
|
||||
### Knowledge Base References
|
||||
|
||||
- `risk-governance.md` - Risk classification framework
|
||||
- `probability-impact.md` - Risk scoring methodology
|
||||
- `test-levels-framework.md` - Test level selection
|
||||
- `test-priorities-matrix.md` - P0-P3 prioritization
|
||||
|
||||
### Related Documents
|
||||
|
||||
- PRD: {prd_link}
|
||||
- Epic: {epic_link}
|
||||
- Architecture: {arch_link}
|
||||
- Tech Spec: {tech_spec_link}
|
||||
|
||||
---
|
||||
|
||||
**Generated by**: BMad TEA Agent - Test Architect Module
|
||||
**Workflow**: `bmad/bmm/testarch/test-design`
|
||||
**Version**: 4.0 (BMad v6)
|
||||
52
bmad/bmm/workflows/testarch/test-design/workflow.yaml
Normal file
52
bmad/bmm/workflows/testarch/test-design/workflow.yaml
Normal file
@@ -0,0 +1,52 @@
|
||||
# Test Architect workflow: test-design
|
||||
name: testarch-test-design
|
||||
description: "Plan risk mitigation and test coverage strategy before development with risk assessment and prioritization"
|
||||
author: "BMad"
|
||||
|
||||
# Critical variables from config
|
||||
config_source: "{project-root}/bmad/bmm/config.yaml"
|
||||
output_folder: "{config_source}:output_folder"
|
||||
user_name: "{config_source}:user_name"
|
||||
communication_language: "{config_source}:communication_language"
|
||||
document_output_language: "{config_source}:document_output_language"
|
||||
date: system-generated
|
||||
|
||||
# Workflow components
|
||||
installed_path: "{project-root}/bmad/bmm/workflows/testarch/test-design"
|
||||
instructions: "{installed_path}/instructions.md"
|
||||
validation: "{installed_path}/checklist.md"
|
||||
template: "{installed_path}/test-design-template.md"
|
||||
|
||||
# Variables and inputs
|
||||
variables:
|
||||
design_level: "full" # full, targeted, minimal - scope of design effort
|
||||
|
||||
# Output configuration
|
||||
default_output_file: "{output_folder}/test-design-epic-{epic_num}.md"
|
||||
|
||||
# Required tools
|
||||
required_tools:
|
||||
- read_file # Read PRD, epics, stories, architecture docs
|
||||
- write_file # Create test design document
|
||||
- list_files # Find related documentation
|
||||
- search_repo # Search for existing tests and patterns
|
||||
|
||||
# Recommended inputs
|
||||
recommended_inputs:
|
||||
- prd: "Product Requirements Document for context"
|
||||
- epics: "Epic documentation (epics.md or specific epic)"
|
||||
- story: "Story markdown with acceptance criteria"
|
||||
- architecture: "Architecture documents (architecture.md, tech-spec)"
|
||||
- existing_tests: "Current test coverage for gap analysis"
|
||||
|
||||
tags:
|
||||
- qa
|
||||
- planning
|
||||
- test-architect
|
||||
- risk-assessment
|
||||
- coverage
|
||||
|
||||
execution_hints:
|
||||
interactive: false # Minimize prompts
|
||||
autonomous: true # Proceed without user input unless blocked
|
||||
iterative: true
|
||||
Reference in New Issue
Block a user