470 lines
12 KiB
Markdown
470 lines
12 KiB
Markdown
# Non-Functional Requirements Assessment Workflow
|
|
|
|
**Workflow ID:** `testarch-nfr`
|
|
**Agent:** Test Architect (TEA)
|
|
**Command:** `bmad tea *nfr-assess`
|
|
|
|
---
|
|
|
|
## Overview
|
|
|
|
The **nfr-assess** workflow performs a comprehensive assessment of non-functional requirements (NFRs) to validate that the implementation meets performance, security, reliability, and maintainability standards before release. It uses evidence-based validation with deterministic PASS/CONCERNS/FAIL rules and provides actionable recommendations for remediation.
|
|
|
|
**Key Features:**
|
|
|
|
- Assess multiple NFR categories (performance, security, reliability, maintainability, custom)
|
|
- Validate NFRs against defined thresholds from tech specs, PRD, or defaults
|
|
- Classify status deterministically (PASS/CONCERNS/FAIL) based on evidence
|
|
- Never guess thresholds - mark as CONCERNS if unknown
|
|
- Generate CI/CD-ready YAML snippets for quality gates
|
|
- Provide quick wins and recommended actions for remediation
|
|
- Create evidence checklists for gaps
|
|
|
|
---
|
|
|
|
## When to Use This Workflow
|
|
|
|
Use `*nfr-assess` when you need to:
|
|
|
|
- ✅ Validate non-functional requirements before release
|
|
- ✅ Assess performance against defined thresholds
|
|
- ✅ Verify security requirements are met
|
|
- ✅ Validate reliability and error handling
|
|
- ✅ Check maintainability standards (coverage, quality, documentation)
|
|
- ✅ Generate NFR assessment reports for stakeholders
|
|
- ✅ Create gate-ready metrics for CI/CD pipelines
|
|
|
|
**Typical Timing:**
|
|
|
|
- Before release (validate all NFRs)
|
|
- Before PR merge (validate critical NFRs)
|
|
- During sprint retrospectives (assess maintainability)
|
|
- After performance testing (validate performance NFRs)
|
|
- After security audit (validate security NFRs)
|
|
|
|
---
|
|
|
|
## Prerequisites
|
|
|
|
**Required:**
|
|
|
|
- Implementation deployed locally or accessible for evaluation
|
|
- Evidence sources available (test results, metrics, logs, CI results)
|
|
|
|
**Recommended:**
|
|
|
|
- NFR requirements defined in tech-spec.md, PRD.md, or story
|
|
- Test results from performance, security, reliability tests
|
|
- Application metrics (response times, error rates, throughput)
|
|
- CI/CD pipeline results for burn-in validation
|
|
|
|
**Halt Conditions:**
|
|
|
|
- NFR targets are undefined and cannot be obtained → Halt and request definition
|
|
- Implementation is not accessible for evaluation → Halt and request deployment
|
|
|
|
---
|
|
|
|
## Usage
|
|
|
|
### Basic Usage (BMad Mode)
|
|
|
|
```bash
|
|
bmad tea *nfr-assess
|
|
```
|
|
|
|
The workflow will:
|
|
|
|
1. Read tech-spec.md for NFR requirements
|
|
2. Gather evidence from test results, metrics, logs
|
|
3. Assess each NFR category against thresholds
|
|
4. Generate NFR assessment report
|
|
5. Save to `bmad/output/nfr-assessment.md`
|
|
|
|
### Standalone Mode (No Tech Spec)
|
|
|
|
```bash
|
|
bmad tea *nfr-assess --feature-name "User Authentication"
|
|
```
|
|
|
|
### Custom Configuration
|
|
|
|
```bash
|
|
bmad tea *nfr-assess \
|
|
--assess-performance true \
|
|
--assess-security true \
|
|
--assess-reliability true \
|
|
--assess-maintainability true \
|
|
--performance-response-time-ms 500 \
|
|
--security-score-min 85
|
|
```
|
|
|
|
---
|
|
|
|
## Workflow Steps
|
|
|
|
1. **Load Context** - Read tech spec, PRD, knowledge base fragments
|
|
2. **Identify NFRs** - Determine categories and thresholds
|
|
3. **Gather Evidence** - Read test results, metrics, logs, CI results
|
|
4. **Assess NFRs** - Apply deterministic PASS/CONCERNS/FAIL rules
|
|
5. **Identify Actions** - Quick wins, recommended actions, monitoring hooks
|
|
6. **Generate Deliverables** - NFR assessment report, gate YAML, evidence checklist
|
|
|
|
---
|
|
|
|
## Outputs
|
|
|
|
### NFR Assessment Report (`nfr-assessment.md`)
|
|
|
|
Comprehensive markdown file with:
|
|
|
|
- Executive summary (overall status, critical issues)
|
|
- Assessment by category (performance, security, reliability, maintainability)
|
|
- Evidence for each NFR (test results, metrics, thresholds)
|
|
- Status classification (PASS/CONCERNS/FAIL)
|
|
- Quick wins section
|
|
- Recommended actions section
|
|
- Evidence gaps checklist
|
|
|
|
### Gate YAML Snippet (Optional)
|
|
|
|
```yaml
|
|
nfr_assessment:
|
|
date: '2025-10-14'
|
|
categories:
|
|
performance: 'PASS'
|
|
security: 'CONCERNS'
|
|
reliability: 'PASS'
|
|
maintainability: 'PASS'
|
|
overall_status: 'CONCERNS'
|
|
critical_issues: 0
|
|
high_priority_issues: 1
|
|
concerns: 1
|
|
blockers: false
|
|
```
|
|
|
|
### Evidence Checklist (Optional)
|
|
|
|
- List of NFRs with missing or incomplete evidence
|
|
- Owners for evidence collection
|
|
- Suggested evidence sources
|
|
- Deadlines for evidence collection
|
|
|
|
---
|
|
|
|
## NFR Categories
|
|
|
|
### Performance
|
|
|
|
**Criteria:** Response time, throughput, resource usage, scalability
|
|
**Thresholds (Default):**
|
|
|
|
- Response time p95: 500ms
|
|
- Throughput: 100 RPS
|
|
- CPU usage: < 70%
|
|
- Memory usage: < 80%
|
|
|
|
**Evidence Sources:** Load test results, APM data, Lighthouse reports, Playwright traces
|
|
|
|
---
|
|
|
|
### Security
|
|
|
|
**Criteria:** Authentication, authorization, data protection, vulnerability management
|
|
**Thresholds (Default):**
|
|
|
|
- Security score: >= 85/100
|
|
- Critical vulnerabilities: 0
|
|
- High vulnerabilities: < 3
|
|
- MFA enabled
|
|
|
|
**Evidence Sources:** SAST results, DAST results, dependency scanning, pentest reports
|
|
|
|
---
|
|
|
|
### Reliability
|
|
|
|
**Criteria:** Availability, error handling, fault tolerance, disaster recovery
|
|
**Thresholds (Default):**
|
|
|
|
- Uptime: >= 99.9%
|
|
- Error rate: < 0.1%
|
|
- MTTR: < 15 minutes
|
|
- CI burn-in: 100 consecutive runs
|
|
|
|
**Evidence Sources:** Uptime monitoring, error logs, CI burn-in results, chaos tests
|
|
|
|
---
|
|
|
|
### Maintainability
|
|
|
|
**Criteria:** Code quality, test coverage, documentation, technical debt
|
|
**Thresholds (Default):**
|
|
|
|
- Test coverage: >= 80%
|
|
- Code quality: >= 85/100
|
|
- Technical debt: < 5%
|
|
- Documentation: >= 90%
|
|
|
|
**Evidence Sources:** Coverage reports, static analysis, documentation audit, test review
|
|
|
|
---
|
|
|
|
## Assessment Rules
|
|
|
|
### PASS ✅
|
|
|
|
- Evidence exists AND meets or exceeds threshold
|
|
- No concerns flagged in evidence
|
|
- Quality is acceptable
|
|
|
|
### CONCERNS ⚠️
|
|
|
|
- Threshold is UNKNOWN (not defined)
|
|
- Evidence is MISSING or INCOMPLETE
|
|
- Evidence is close to threshold (within 10%)
|
|
- Evidence shows intermittent issues
|
|
|
|
### FAIL ❌
|
|
|
|
- Evidence exists BUT does not meet threshold
|
|
- Critical evidence is MISSING
|
|
- Evidence shows consistent failures
|
|
- Quality is unacceptable
|
|
|
|
---
|
|
|
|
## Configuration
|
|
|
|
### workflow.yaml Variables
|
|
|
|
```yaml
|
|
variables:
|
|
# NFR categories to assess
|
|
assess_performance: true
|
|
assess_security: true
|
|
assess_reliability: true
|
|
assess_maintainability: true
|
|
|
|
# Custom NFR categories
|
|
custom_nfr_categories: '' # e.g., "accessibility,compliance"
|
|
|
|
# Evidence sources
|
|
test_results_dir: '{project-root}/test-results'
|
|
metrics_dir: '{project-root}/metrics'
|
|
logs_dir: '{project-root}/logs'
|
|
include_ci_results: true
|
|
|
|
# Thresholds
|
|
performance_response_time_ms: 500
|
|
performance_throughput_rps: 100
|
|
security_score_min: 85
|
|
reliability_uptime_pct: 99.9
|
|
maintainability_coverage_pct: 80
|
|
|
|
# Assessment configuration
|
|
use_deterministic_rules: true
|
|
never_guess_thresholds: true
|
|
require_evidence: true
|
|
suggest_monitoring: true
|
|
|
|
# Output configuration
|
|
output_file: '{output_folder}/nfr-assessment.md'
|
|
generate_gate_yaml: true
|
|
generate_evidence_checklist: true
|
|
```
|
|
|
|
---
|
|
|
|
## Knowledge Base Integration
|
|
|
|
This workflow automatically loads relevant knowledge fragments:
|
|
|
|
- `nfr-criteria.md` - Non-functional requirements criteria
|
|
- `ci-burn-in.md` - CI/CD burn-in patterns for reliability
|
|
- `test-quality.md` - Test quality expectations (maintainability)
|
|
- `playwright-config.md` - Performance configuration patterns
|
|
|
|
---
|
|
|
|
## Examples
|
|
|
|
### Example 1: Full NFR Assessment Before Release
|
|
|
|
```bash
|
|
bmad tea *nfr-assess
|
|
```
|
|
|
|
**Output:**
|
|
|
|
```markdown
|
|
# NFR Assessment - Story 1.3
|
|
|
|
**Overall Status:** PASS ✅ (No blockers)
|
|
|
|
## Performance Assessment
|
|
|
|
- Response Time p95: PASS ✅ (320ms < 500ms threshold)
|
|
- Throughput: PASS ✅ (250 RPS > 100 RPS threshold)
|
|
|
|
## Security Assessment
|
|
|
|
- Authentication: PASS ✅ (MFA enforced)
|
|
- Data Protection: PASS ✅ (AES-256 + TLS 1.3)
|
|
|
|
## Reliability Assessment
|
|
|
|
- Uptime: PASS ✅ (99.95% > 99.9% threshold)
|
|
- Error Rate: PASS ✅ (0.05% < 0.1% threshold)
|
|
|
|
## Maintainability Assessment
|
|
|
|
- Test Coverage: PASS ✅ (87% > 80% threshold)
|
|
- Code Quality: PASS ✅ (92/100 > 85/100 threshold)
|
|
|
|
Gate Status: PASS ✅ - Ready for release
|
|
```
|
|
|
|
### Example 2: NFR Assessment with Concerns
|
|
|
|
```bash
|
|
bmad tea *nfr-assess --feature-name "User Authentication"
|
|
```
|
|
|
|
**Output:**
|
|
|
|
```markdown
|
|
# NFR Assessment - User Authentication
|
|
|
|
**Overall Status:** CONCERNS ⚠️ (1 HIGH issue)
|
|
|
|
## Security Assessment
|
|
|
|
### Authentication Strength
|
|
|
|
- **Status:** CONCERNS ⚠️
|
|
- **Threshold:** MFA enabled for all users
|
|
- **Actual:** MFA optional (not enforced)
|
|
- **Evidence:** Security audit (security-audit-2025-10-14.md)
|
|
- **Recommendation:** HIGH - Enforce MFA for all new accounts
|
|
|
|
## Quick Wins
|
|
|
|
1. **Enforce MFA (Security)** - HIGH - 4 hours
|
|
- Add configuration flag to enforce MFA
|
|
- No code changes needed
|
|
|
|
Gate Status: CONCERNS ⚠️ - Address HIGH priority issues before release
|
|
```
|
|
|
|
### Example 3: Performance-Only Assessment
|
|
|
|
```bash
|
|
bmad tea *nfr-assess \
|
|
--assess-performance true \
|
|
--assess-security false \
|
|
--assess-reliability false \
|
|
--assess-maintainability false
|
|
```
|
|
|
|
---
|
|
|
|
## Troubleshooting
|
|
|
|
### "NFR thresholds not defined"
|
|
|
|
- Check tech-spec.md for NFR requirements
|
|
- Check PRD.md for product-level SLAs
|
|
- Check story file for feature-specific requirements
|
|
- If thresholds truly unknown, mark as CONCERNS and recommend defining them
|
|
|
|
### "No evidence found"
|
|
|
|
- Check evidence directories (test-results, metrics, logs)
|
|
- Check CI/CD pipeline for test results
|
|
- If evidence truly missing, mark NFR as "NO EVIDENCE" and recommend generating it
|
|
|
|
### "CONCERNS status but no threshold exceeded"
|
|
|
|
- CONCERNS is correct when threshold is UNKNOWN or evidence is MISSING/INCOMPLETE
|
|
- CONCERNS is also correct when evidence is close to threshold (within 10%)
|
|
- Document why CONCERNS was assigned in assessment report
|
|
|
|
### "FAIL status blocks release"
|
|
|
|
- This is intentional - FAIL means critical NFR not met
|
|
- Recommend remediation actions with specific steps
|
|
- Re-run assessment after remediation
|
|
|
|
---
|
|
|
|
## Integration with Other Workflows
|
|
|
|
- **testarch-test-design** → `*nfr-assess` - Define NFR requirements, then assess
|
|
- **testarch-framework** → `*nfr-assess` - Set up frameworks, then validate NFRs
|
|
- **testarch-ci** → `*nfr-assess` - Configure CI, then assess reliability with burn-in
|
|
- `*nfr-assess` → **testarch-trace (Phase 2)** - Assess NFRs, then apply quality gates
|
|
- `*nfr-assess` → **testarch-test-review** - Assess maintainability, then review tests
|
|
|
|
---
|
|
|
|
## Best Practices
|
|
|
|
1. **Never Guess Thresholds**
|
|
- If threshold is unknown, mark as CONCERNS
|
|
- Recommend defining threshold in tech-spec.md
|
|
- Don't infer thresholds from similar features
|
|
|
|
2. **Evidence-Based Assessment**
|
|
- Every assessment must be backed by evidence
|
|
- Mark NFRs without evidence as "NO EVIDENCE"
|
|
- Don't assume or infer - require explicit evidence
|
|
|
|
3. **Deterministic Rules**
|
|
- Apply PASS/CONCERNS/FAIL consistently
|
|
- Document reasoning for each classification
|
|
- Use same rules across all NFR categories
|
|
|
|
4. **Actionable Recommendations**
|
|
- Provide specific steps, not generic advice
|
|
- Include priority, effort estimate, owner suggestion
|
|
- Focus on quick wins first
|
|
|
|
5. **Gate Integration**
|
|
- Enable `generate_gate_yaml` for CI/CD integration
|
|
- Use YAML snippets in pipeline quality gates
|
|
- Export metrics for dashboard visualization
|
|
|
|
---
|
|
|
|
## Quality Gates
|
|
|
|
| Status | Criteria | Action |
|
|
| ----------- | ---------------------------- | --------------------------- |
|
|
| PASS ✅ | All NFRs have PASS status | Ready for release |
|
|
| CONCERNS ⚠️ | Any NFR has CONCERNS status | Address before next release |
|
|
| FAIL ❌ | Critical NFR has FAIL status | Do not release - BLOCKER |
|
|
|
|
---
|
|
|
|
## Related Commands
|
|
|
|
- `bmad tea *test-design` - Define NFR requirements and test plan
|
|
- `bmad tea *framework` - Set up performance/security testing frameworks
|
|
- `bmad tea *ci` - Configure CI/CD for NFR validation
|
|
- `bmad tea *trace` (Phase 2) - Apply quality gates using NFR assessment metrics
|
|
- `bmad tea *test-review` - Review test quality (maintainability NFR)
|
|
|
|
---
|
|
|
|
## Resources
|
|
|
|
- [Instructions](./instructions.md) - Detailed workflow steps
|
|
- [Checklist](./checklist.md) - Validation checklist
|
|
- [Template](./nfr-report-template.md) - NFR assessment report template
|
|
- [Knowledge Base](../../testarch/knowledge/) - NFR criteria and best practices
|
|
|
|
---
|
|
|
|
<!-- Powered by BMAD-CORE™ -->
|