bmad初始化

This commit is contained in:
2025-11-01 19:22:39 +08:00
parent 5b21dc0bd5
commit 426ae41f54
447 changed files with 80633 additions and 0 deletions

View File

@@ -0,0 +1,869 @@
# Automate Workflow
Expands test automation coverage by generating comprehensive test suites at appropriate levels (E2E, API, Component, Unit) with supporting infrastructure. This workflow operates in **dual mode** - works seamlessly WITH or WITHOUT BMad artifacts.
**Core Principle**: Generate prioritized, deterministic tests that avoid duplicate coverage and follow testing best practices.
## Usage
```bash
bmad tea *automate
```
The TEA agent runs this workflow when:
- **BMad-Integrated**: After story implementation to expand coverage beyond ATDD tests
- **Standalone**: Point at any codebase/feature and generate tests independently ("work out of thin air")
- **Auto-discover**: No targets specified - scans codebase for features needing tests
## Inputs
**Execution Modes:**
1. **BMad-Integrated Mode** (story available) - OPTIONAL
2. **Standalone Mode** (no BMad artifacts) - Direct code analysis
3. **Auto-discover Mode** (no targets) - Scan for coverage gaps
**Required Context Files:**
- **Framework configuration**: Test framework config (playwright.config.ts or cypress.config.ts) - REQUIRED
**Optional Context (BMad-Integrated Mode):**
- **Story markdown** (`{story_file}`): User story with acceptance criteria (enhances coverage targeting but NOT required)
- **Tech spec**: Technical specification (provides architectural context)
- **Test design**: Risk/priority context (P0-P3 alignment)
- **PRD**: Product requirements (business context)
**Optional Context (Standalone Mode):**
- **Source code**: Feature implementation to analyze
- **Existing tests**: Current test suite for gap analysis
**Workflow Variables:**
- `standalone_mode`: Can work without BMad artifacts (default: true)
- `story_file`: Path to story markdown (optional)
- `target_feature`: Feature name or directory to analyze (e.g., "user-authentication" or "src/auth/")
- `target_files`: Specific files to analyze (comma-separated paths)
- `test_dir`: Directory for test files (default: `{project-root}/tests`)
- `source_dir`: Source code directory (default: `{project-root}/src`)
- `auto_discover_features`: Automatically find features needing tests (default: true)
- `analyze_coverage`: Check existing test coverage gaps (default: true)
- `coverage_target`: Coverage strategy - "critical-paths", "comprehensive", "selective" (default: "critical-paths")
- `test_levels`: Which levels to generate - "e2e,api,component,unit" (default: all)
- `avoid_duplicate_coverage`: Don't test same behavior at multiple levels (default: true)
- `include_p0`: Include P0 critical path tests (default: true)
- `include_p1`: Include P1 high priority tests (default: true)
- `include_p2`: Include P2 medium priority tests (default: true)
- `include_p3`: Include P3 low priority tests (default: false)
- `use_given_when_then`: BDD-style test structure (default: true)
- `one_assertion_per_test`: Atomic test design (default: true)
- `network_first`: Route interception before navigation (default: true)
- `deterministic_waits`: No hard waits or sleeps (default: true)
- `generate_fixtures`: Create/enhance fixture architecture (default: true)
- `generate_factories`: Create/enhance data factories (default: true)
- `update_helpers`: Add utility functions (default: true)
- `use_test_design`: Load test-design.md if exists (default: true)
- `use_tech_spec`: Load tech-spec.md if exists (default: true)
- `use_prd`: Load PRD.md if exists (default: true)
- `update_readme`: Update test README with new specs (default: true)
- `update_package_scripts`: Add test execution scripts (default: true)
- `output_summary`: Path for automation summary (default: `{output_folder}/automation-summary.md`)
- `max_test_duration`: Maximum seconds per test (default: 90)
- `max_file_lines`: Maximum lines per test file (default: 300)
- `require_self_cleaning`: All tests must clean up data (default: true)
- `auto_load_knowledge`: Load relevant knowledge fragments (default: true)
- `run_tests_after_generation`: Verify tests pass/fail as expected (default: true)
- `auto_validate`: Run generated tests after creation (default: true) **NEW**
- `auto_heal_failures`: Enable automatic healing (default: false, opt-in) **NEW**
- `max_healing_iterations`: Maximum healing attempts per test (default: 3) **NEW**
- `fail_on_unhealable`: Fail workflow if tests can't be healed (default: false) **NEW**
- `mark_unhealable_as_fixme`: Mark unfixable tests with test.fixme() (default: true) **NEW**
- `use_mcp_healing`: Use Playwright MCP if available (default: true) **NEW**
- `healing_knowledge_fragments`: Healing patterns to load (default: "test-healing-patterns,selector-resilience,timing-debugging") **NEW**
## Outputs
**Primary Deliverable:**
- **Automation Summary** (`automation-summary.md`): Comprehensive report containing:
- Execution mode (BMad-Integrated, Standalone, Auto-discover)
- Feature analysis (source files analyzed, coverage gaps)
- Tests created (E2E, API, Component, Unit) with counts and paths
- Infrastructure created (fixtures, factories, helpers)
- Test execution instructions
- Coverage analysis (P0-P3 breakdown, coverage percentage)
- Definition of Done checklist
- Next steps and recommendations
**Test Files Created:**
- **E2E tests** (`tests/e2e/{feature-name}.spec.ts`): Critical user journeys (P0-P1)
- **API tests** (`tests/api/{feature-name}.api.spec.ts`): Business logic and contracts (P1-P2)
- **Component tests** (`tests/component/{ComponentName}.test.tsx`): UI behavior (P1-P2)
- **Unit tests** (`tests/unit/{module-name}.test.ts`): Pure logic (P2-P3)
**Supporting Infrastructure:**
- **Fixtures** (`tests/support/fixtures/{feature}.fixture.ts`): Setup/teardown with auto-cleanup
- **Data factories** (`tests/support/factories/{entity}.factory.ts`): Random test data using faker
- **Helpers** (`tests/support/helpers/{utility}.ts`): Utility functions (waitFor, retry, etc.)
**Documentation Updates:**
- **Test README** (`tests/README.md`): Test suite overview, execution instructions, priority tagging, patterns
- **package.json scripts**: Test execution commands (test:e2e, test:e2e:p0, test:api, etc.)
**Validation Safeguards:**
- All tests follow Given-When-Then format
- All tests have priority tags ([P0], [P1], [P2], [P3])
- All tests use data-testid selectors (stable, not CSS classes)
- All tests are self-cleaning (fixtures with auto-cleanup)
- No hard waits or flaky patterns (deterministic)
- Test files under 300 lines (lean and focused)
- Tests run under 1.5 minutes each (fast feedback)
## Key Features
### Dual-Mode Operation
**BMad-Integrated Mode** (story available):
- Uses story acceptance criteria for coverage targeting
- Aligns with test-design risk/priority assessment
- Expands ATDD tests with edge cases and negative paths
- Optional - story enhances coverage but not required
**Standalone Mode** (no story):
- Analyzes source code independently
- Identifies coverage gaps automatically
- Generates tests based on code analysis
- Works with any project (BMad or non-BMad)
**Auto-discover Mode** (no targets):
- Scans codebase for features needing tests
- Prioritizes features with no coverage
- Generates comprehensive test plan
### Avoid Duplicate Coverage
**Critical principle**: Don't test same behavior at multiple levels
**Good coverage strategy:**
- **E2E**: User can login → Dashboard loads (critical happy path only)
- **API**: POST /auth/login returns correct status codes (variations: 200, 401, 400)
- **Component**: LoginForm validates input (UI edge cases: empty fields, invalid format)
- **Unit**: validateEmail() logic (pure function edge cases)
**Bad coverage (duplicate):**
- E2E: User can login → Dashboard loads
- E2E: User can login with different emails → Dashboard loads (unnecessary duplication)
- API: POST /auth/login returns 200 (already covered in E2E)
Use E2E sparingly for critical paths. Use API/Component/Unit for variations and edge cases.
### Healing Capabilities (NEW - Phase 2.5)
**automate** automatically validates and heals test failures after generation.
**Configuration**: Controlled by `config.tea_use_mcp_enhancements` (default: true)
- If true + MCP available → MCP-assisted healing
- If true + MCP unavailable → Pattern-based healing
- If false → No healing, document failures for manual review
**Constants**: Max 3 healing attempts, unfixable tests marked as `test.fixme()`
**How Healing Works (Default - Pattern-Based):**
TEA heals tests using pattern-based analysis by:
1. **Parsing error messages** from test output logs
2. **Matching patterns** against known failure signatures
3. **Applying fixes** from healing knowledge fragments:
- `test-healing-patterns.md` - Common failure patterns (selectors, timing, data, network)
- `selector-resilience.md` - Selector refactoring (CSS → data-testid, nth() → filter())
- `timing-debugging.md` - Race condition fixes (hard waits → event-based waits)
4. **Re-running tests** to verify fix (max 3 iterations)
5. **Marking unfixable tests** as `test.fixme()` with detailed comments
**This works well for:**
- ✅ Common failure patterns (stale selectors, timing issues, dynamic data)
- ✅ Text-based errors with clear signatures
- ✅ Issues documented in knowledge base
- ✅ Automated CI environments without browser access
**What MCP Adds (Interactive Debugging Enhancement):**
When Playwright MCP is available, TEA **additionally**:
1. **Debugs failures interactively** before applying pattern-based fixes:
- **Pause test execution** with `playwright_test_debug_test` (step through, inspect state)
- **See visual failure context** with `browser_snapshot` (screenshot of failure state)
- **Inspect live DOM** with browser tools (find why selector doesn't match)
- **Analyze console logs** with `browser_console_messages` (JS errors, warnings, debug output)
- **Inspect network activity** with `browser_network_requests` (failed API calls, CORS errors, timeouts)
2. **Enhances pattern-based fixes** with real-world data:
- **Pattern match identifies issue** (e.g., "stale selector")
- **MCP discovers actual selector** with `browser_generate_locator` from live page
- **TEA applies refined fix** using real DOM structure (not just pattern guess)
- **Verification happens in browser** (see if fix works visually)
3. **Catches root causes** pattern matching might miss:
- **Network failures**: MCP shows 500 error on API call (not just timeout)
- **JS errors**: MCP shows `TypeError: undefined` in console (not just "element not found")
- **Timing issues**: MCP shows loading spinner still visible (not just "selector timeout")
- **State problems**: MCP shows modal blocking button (not just "not clickable")
**Key Benefits of MCP Enhancement:**
-**Pattern-based fixes** (fast, automated) **+** **MCP verification** (accurate, context-aware)
-**Visual debugging**: See exactly what user sees when test fails
-**DOM inspection**: Discover why selectors don't match (element missing, wrong attributes, dynamic IDs)
-**Network visibility**: Identify API failures, slow requests, CORS issues
-**Console analysis**: Catch JS errors that break page functionality
-**Robust selectors**: Generate locators from actual DOM (role, text, testid hierarchy)
-**Faster iteration**: Debug and fix in same browser session (no restart needed)
-**Higher success rate**: MCP helps diagnose failures pattern matching can't solve
**Example Enhancement Flow:**
```
1. Pattern-based healing identifies issue
→ Error: "Locator '.submit-btn' resolved to 0 elements"
→ Pattern match: Stale selector (CSS class)
→ Suggested fix: Replace with data-testid
2. MCP enhances diagnosis (if available)
→ browser_snapshot shows button exists but has class ".submit-button" (not ".submit-btn")
→ browser_generate_locator finds: button[type="submit"].submit-button
→ browser_console_messages shows no errors
3. TEA applies refined fix
→ await page.locator('button[type="submit"]').click()
→ (More accurate than pattern-based guess)
```
**Healing Modes:**
1. **MCP-Enhanced Healing** (when Playwright MCP available):
- Pattern-based analysis **+** Interactive debugging
- Visual context with `browser_snapshot`
- Console log analysis with `browser_console_messages`
- Network inspection with `browser_network_requests`
- Live DOM inspection with `browser_generate_locator`
- Step-by-step debugging with `playwright_test_debug_test`
2. **Pattern-Based Healing** (always available):
- Error message parsing and pattern matching
- Automated fixes from healing knowledge fragments
- Text-based analysis (no visual/DOM inspection)
- Works in CI without browser access
**Healing Workflow:**
```
1. Generate tests → Run tests
2. IF pass → Success ✅
3. IF fail AND auto_heal_failures=false → Report failures ⚠️
4. IF fail AND auto_heal_failures=true → Enter healing loop:
a. Identify failure pattern (selector, timing, data, network)
b. Apply automated fix from knowledge base
c. Re-run test (max 3 iterations)
d. IF healed → Success ✅
e. IF unhealable → Mark test.fixme() with detailed comment
```
**Example Healing Outcomes:**
```typescript
// ❌ Original (failing): CSS class selector
await page.locator('.btn-primary').click();
// ✅ Healed: data-testid selector
await page.getByTestId('submit-button').click();
// ❌ Original (failing): Hard wait
await page.waitForTimeout(3000);
// ✅ Healed: Network-first pattern
await page.waitForResponse('**/api/data');
// ❌ Original (failing): Hardcoded ID
await expect(page.getByText('User 123')).toBeVisible();
// ✅ Healed: Regex pattern
await expect(page.getByText(/User \d+/)).toBeVisible();
```
**Unfixable Tests (Marked as test.fixme()):**
```typescript
test.fixme('[P1] should handle complex interaction', async ({ page }) => {
// FIXME: Test healing failed after 3 attempts
// Failure: "Locator 'button[data-action="submit"]' resolved to 0 elements"
// Attempted fixes:
// 1. Replaced with page.getByTestId('submit-button') - still failing
// 2. Replaced with page.getByRole('button', { name: 'Submit' }) - still failing
// 3. Added waitForLoadState('networkidle') - still failing
// Manual investigation needed: Selector may require application code changes
// TODO: Review with team, may need data-testid added to button component
// Original test code...
});
```
**When to Enable Healing:**
- ✅ Enable for greenfield projects (catch generated test issues early)
- ✅ Enable for brownfield projects (auto-fix legacy selector patterns)
- ❌ Disable if environment not ready (application not deployed/seeded)
- ❌ Disable if preferring manual review of all generated tests
**Healing Report Example:**
```markdown
## Test Healing Report
**Auto-Heal Enabled**: true
**Healing Mode**: Pattern-based
**Iterations Allowed**: 3
### Validation Results
- **Total tests**: 10
- **Passing**: 7
- **Failing**: 3
### Healing Outcomes
**Successfully Healed (2 tests):**
- `tests/e2e/login.spec.ts:15` - Stale selector (CSS class → data-testid)
- `tests/e2e/checkout.spec.ts:42` - Race condition (added network-first interception)
**Unable to Heal (1 test):**
- `tests/e2e/complex-flow.spec.ts:67` - Marked as test.fixme()
- Requires application code changes (add data-testid to component)
### Healing Patterns Applied
- **Selector fixes**: 1
- **Timing fixes**: 1
```
**Graceful Degradation:**
- Healing is OPTIONAL (default: disabled)
- Works without Playwright MCP (pattern-based fallback)
- Unfixable tests marked clearly (not silently broken)
- Manual investigation path documented
### Recording Mode (NEW - Phase 2.5)
**automate** can record complex UI interactions instead of AI generation.
**Activation**: Automatic for complex UI scenarios when config.tea_use_mcp_enhancements is true and MCP available
- Complex scenarios: drag-drop, wizards, multi-page flows
- Fallback: AI generation (silent, automatic)
**When to Use Recording Mode:**
- ✅ Complex UI interactions (drag-drop, multi-step forms, wizards)
- ✅ Visual workflows (modals, dialogs, animations, transitions)
- ✅ Unclear requirements (exploratory, discovering behavior)
- ✅ Multi-page flows (checkout, registration, onboarding)
- ❌ NOT for simple CRUD (AI generation faster)
- ❌ NOT for API-only tests (no UI to record)
**When to Use AI Generation (Default):**
- ✅ Clear requirements available
- ✅ Standard patterns (login, CRUD, navigation)
- ✅ Need many tests quickly
- ✅ API/backend tests (no UI interaction)
**Recording Workflow (Same as atdd):**
```
1. Set generation_mode: "recording"
2. Use generator_setup_page to init recording
3. For each test scenario:
- Execute with browser_* tools (navigate, click, type, select)
- Add verifications with browser_verify_* tools
- Capture log and generate test file
4. Enhance with knowledge base patterns:
- Given-When-Then structure
- data-testid selectors
- Network-first interception
- Fixtures/factories
5. Validate (run tests if auto_validate enabled)
6. Heal if needed (if auto_heal_failures enabled)
```
**Combination: Recording + Healing:**
automate can use BOTH recording and healing together:
- Generate tests via recording (complex flows captured interactively)
- Run tests to validate (auto_validate)
- Heal failures automatically (auto_heal_failures)
This is particularly powerful for brownfield projects where:
- Requirements unclear → Use recording to capture existing behavior
- Application complex → Recording captures nuances AI might miss
- Tests may fail → Healing fixes common issues automatically
**Graceful Degradation:**
- Recording mode is OPTIONAL (default: AI generation)
- Requires Playwright MCP (falls back to AI if unavailable)
- Works with or without healing enabled
- Same quality output regardless of generation method
### Test Level Selection Framework
**E2E (End-to-End)**:
- Critical user journeys (login, checkout, core workflows)
- Multi-system integration
- User-facing acceptance criteria
- Characteristics: High confidence, slow execution, brittle
**API (Integration)**:
- Business logic validation
- Service contracts and data transformations
- Backend integration without UI
- Characteristics: Fast feedback, good balance, stable
**Component**:
- UI component behavior (buttons, forms, modals)
- Interaction testing (click, hover, keyboard navigation)
- State management within component
- Characteristics: Fast, isolated, granular
**Unit**:
- Pure business logic and algorithms
- Edge cases and error handling
- Minimal dependencies
- Characteristics: Fastest, most granular
### Priority Classification (P0-P3)
**P0 (Critical - Every commit)**:
- Critical user paths that must always work
- Security-critical functionality (auth, permissions)
- Data integrity scenarios
- Run in pre-commit hooks or PR checks
**P1 (High - PR to main)**:
- Important features with high user impact
- Integration points between systems
- Error handling for common failures
- Run before merging to main branch
**P2 (Medium - Nightly)**:
- Edge cases with moderate impact
- Less-critical feature variations
- Performance/load testing
- Run in nightly CI builds
**P3 (Low - On-demand)**:
- Nice-to-have validations
- Rarely-used features
- Exploratory testing scenarios
- Run manually or weekly
**Priority tagging enables selective execution:**
```bash
npm run test:e2e:p0 # Run only P0 tests (critical paths)
npm run test:e2e:p1 # Run P0 + P1 tests (pre-merge)
```
### Given-When-Then Test Structure
All tests follow BDD format for clarity:
```typescript
test('[P0] should login with valid credentials and load dashboard', async ({ page }) => {
// GIVEN: User is on login page
await page.goto('/login');
// WHEN: User submits valid credentials
await page.fill('[data-testid="email-input"]', 'user@example.com');
await page.fill('[data-testid="password-input"]', 'Password123!');
await page.click('[data-testid="login-button"]');
// THEN: User is redirected to dashboard
await expect(page).toHaveURL('/dashboard');
await expect(page.locator('[data-testid="user-name"]')).toBeVisible();
});
```
### One Assertion Per Test (Atomic Design)
Each test verifies exactly one behavior:
```typescript
// ✅ CORRECT: One assertion
test('[P0] should display user name', async ({ page }) => {
await expect(page.locator('[data-testid="user-name"]')).toHaveText('John');
});
// ❌ WRONG: Multiple assertions (not atomic)
test('[P0] should display user info', async ({ page }) => {
await expect(page.locator('[data-testid="user-name"]')).toHaveText('John');
await expect(page.locator('[data-testid="user-email"]')).toHaveText('john@example.com');
});
```
**Why?** If second assertion fails, you don't know if first is still valid. Split into separate tests for clear failure diagnosis.
### Network-First Testing Pattern
**Critical pattern to prevent race conditions**:
```typescript
test('should load user dashboard after login', async ({ page }) => {
// CRITICAL: Intercept routes BEFORE navigation
await page.route('**/api/user', (route) =>
route.fulfill({
status: 200,
body: JSON.stringify({ id: 1, name: 'Test User' }),
}),
);
// NOW navigate
await page.goto('/dashboard');
await expect(page.locator('[data-testid="user-name"]')).toHaveText('Test User');
});
```
Always set up route interception before navigating to pages that make network requests.
### Fixture Architecture with Auto-Cleanup
Playwright fixtures with automatic data cleanup:
```typescript
// tests/support/fixtures/auth.fixture.ts
import { test as base } from '@playwright/test';
import { createUser, deleteUser } from '../factories/user.factory';
export const test = base.extend({
authenticatedUser: async ({ page }, use) => {
// Setup: Create and authenticate user
const user = await createUser();
await page.goto('/login');
await page.fill('[data-testid="email"]', user.email);
await page.fill('[data-testid="password"]', user.password);
await page.click('[data-testid="login-button"]');
await page.waitForURL('/dashboard');
// Provide to test
await use(user);
// Cleanup: Delete user automatically
await deleteUser(user.id);
},
});
```
**Fixture principles:**
- Auto-cleanup (always delete created data in teardown)
- Composable (fixtures can use other fixtures)
- Isolated (each test gets fresh data)
- Type-safe with TypeScript
### Data Factory Architecture
Use faker for all test data generation:
```typescript
// tests/support/factories/user.factory.ts
import { faker } from '@faker-js/faker';
export const createUser = (overrides = {}) => ({
id: faker.number.int(),
email: faker.internet.email(),
password: faker.internet.password(),
name: faker.person.fullName(),
role: 'user',
createdAt: faker.date.recent().toISOString(),
...overrides,
});
export const createUsers = (count: number) => Array.from({ length: count }, () => createUser());
// API helper for cleanup
export const deleteUser = async (userId: number) => {
await fetch(`/api/users/${userId}`, { method: 'DELETE' });
};
```
**Factory principles:**
- Use faker for random data (no hardcoded values to prevent collisions)
- Support overrides for specific test scenarios
- Generate complete valid objects matching API contracts
- Include helper functions for bulk creation and cleanup
### No Page Objects
**Do NOT create page object classes.** Keep tests simple and direct:
```typescript
// ✅ CORRECT: Direct test
test('should login', async ({ page }) => {
await page.goto('/login');
await page.fill('[data-testid="email"]', 'user@example.com');
await page.click('[data-testid="login-button"]');
await expect(page).toHaveURL('/dashboard');
});
// ❌ WRONG: Page object abstraction
class LoginPage {
async login(email, password) { ... }
}
```
Use fixtures for setup/teardown, not page objects for actions.
### Deterministic Tests Only
**No flaky patterns allowed:**
```typescript
// ❌ WRONG: Hard wait
await page.waitForTimeout(2000);
// ✅ CORRECT: Explicit wait
await page.waitForSelector('[data-testid="user-name"]');
await expect(page.locator('[data-testid="user-name"]')).toBeVisible();
// ❌ WRONG: Conditional flow
if (await element.isVisible()) {
await element.click();
}
// ✅ CORRECT: Deterministic assertion
await expect(element).toBeVisible();
await element.click();
// ❌ WRONG: Try-catch for test logic
try {
await element.click();
} catch (e) {
// Test shouldn't catch errors
}
// ✅ CORRECT: Let test fail if element not found
await element.click();
```
## Integration with Other Workflows
**Before this workflow:**
- **framework** workflow: Establish test framework architecture (Playwright/Cypress config, directory structure) - REQUIRED
- **test-design** workflow: Optional for P0-P3 priority alignment and risk assessment context (BMad-Integrated mode only)
- **atdd** workflow: Optional - automate expands beyond ATDD tests with edge cases (BMad-Integrated mode only)
**After this workflow:**
- **trace** workflow: Update traceability matrix with new test coverage (Phase 1) and make quality gate decision (Phase 2)
- **CI pipeline**: Run tests in burn-in loop to detect flaky patterns
**Coordinates with:**
- **DEV agent**: Tests validate implementation correctness
- **Story workflow**: Tests cover acceptance criteria (BMad-Integrated mode only)
## Important Notes
### Works Out of Thin Air
**automate does NOT require BMad artifacts:**
- Can analyze any codebase independently
- User can point TEA at a feature: "automate tests for src/auth/"
- Works on non-BMad projects
- BMad artifacts (story, tech-spec, PRD) are OPTIONAL enhancements, not requirements
**Similar to:**
- **framework**: Can scaffold tests on any project
- **ci**: Can generate CI config without BMad context
**Different from:**
- **atdd**: REQUIRES story with acceptance criteria (halt if missing)
- **test-design**: REQUIRES PRD/epic context (halt if missing)
- **trace (Phase 2)**: REQUIRES test results for gate decision (halt if missing)
### File Size Limits
**Keep test files lean (under 300 lines):**
- If file exceeds limit, split into multiple files by feature area
- Group related tests in describe blocks
- Extract common setup to fixtures
### Quality Standards Enforced
**Every test must:**
- ✅ Use Given-When-Then format
- ✅ Have clear, descriptive name with priority tag
- ✅ One assertion per test (atomic)
- ✅ No hard waits or sleeps
- ✅ Use data-testid selectors (not CSS classes)
- ✅ Self-cleaning (fixtures with auto-cleanup)
- ✅ Deterministic (no flaky patterns)
- ✅ Fast (under 90 seconds)
**Forbidden patterns:**
- ❌ Hard waits: `await page.waitForTimeout(2000)`
- ❌ Conditional flow: `if (await element.isVisible()) { ... }`
- ❌ Try-catch for test logic
- ❌ Hardcoded test data (use factories with faker)
- ❌ Page objects
- ❌ Shared state between tests
## Knowledge Base References
This workflow automatically consults:
- **test-levels-framework.md** - Test level selection (E2E vs API vs Component vs Unit) with characteristics and use cases
- **test-priorities.md** - Priority classification (P0-P3) with execution timing and risk alignment
- **fixture-architecture.md** - Test fixture patterns with setup/teardown and auto-cleanup using Playwright's test.extend()
- **data-factories.md** - Factory patterns using @faker-js/faker for random test data generation with overrides
- **selective-testing.md** - Targeted test execution strategies for CI optimization
- **ci-burn-in.md** - Flaky test detection patterns (10 iterations to catch intermittent failures)
- **test-quality.md** - Test design principles (Given-When-Then, determinism, isolation, atomic assertions)
**Healing Knowledge (If `auto_heal_failures` enabled):**
- **test-healing-patterns.md** - Common failure patterns and automated fixes (selectors, timing, data, network, hard waits)
- **selector-resilience.md** - Robust selector strategies and debugging (data-testid hierarchy, filter vs nth, anti-patterns)
- **timing-debugging.md** - Race condition identification and deterministic wait fixes (network-first, event-based waits)
See `tea-index.csv` for complete knowledge fragment mapping (22 fragments total).
## Example Output
### BMad-Integrated Mode
````markdown
# Automation Summary - User Authentication
**Date:** 2025-10-14
**Story:** Epic 3, Story 5
**Coverage Target:** critical-paths
## Tests Created
### E2E Tests (2 tests, P0-P1)
- `tests/e2e/user-authentication.spec.ts` (87 lines)
- [P0] Login with valid credentials → Dashboard loads
- [P1] Display error for invalid credentials
### API Tests (3 tests, P1-P2)
- `tests/api/auth.api.spec.ts` (102 lines)
- [P1] POST /auth/login - valid credentials → 200 + token
- [P1] POST /auth/login - invalid credentials → 401 + error
- [P2] POST /auth/login - missing fields → 400 + validation
### Component Tests (2 tests, P1)
- `tests/component/LoginForm.test.tsx` (45 lines)
- [P1] Empty fields → submit button disabled
- [P1] Valid input → submit button enabled
## Infrastructure Created
- Fixtures: `tests/support/fixtures/auth.fixture.ts`
- Factories: `tests/support/factories/user.factory.ts`
## Test Execution
```bash
npm run test:e2e # Run all tests
npm run test:e2e:p0 # Critical paths only
npm run test:e2e:p1 # P0 + P1 tests
```
````
## Coverage Analysis
**Total:** 7 tests (P0: 1, P1: 5, P2: 1)
**Levels:** E2E: 2, API: 3, Component: 2
✅ All acceptance criteria covered
✅ Happy path (E2E + API)
✅ Error cases (API)
✅ UI validation (Component)
````
### Standalone Mode
```markdown
# Automation Summary - src/auth/
**Date:** 2025-10-14
**Target:** src/auth/ (standalone analysis)
**Coverage Target:** critical-paths
## Feature Analysis
**Source Files Analyzed:**
- `src/auth/login.ts`
- `src/auth/session.ts`
- `src/auth/validation.ts`
**Existing Coverage:** 0 tests found
**Coverage Gaps:**
- ❌ No E2E tests for login flow
- ❌ No API tests for /auth/login endpoint
- ❌ No unit tests for validateEmail()
## Tests Created
{Same structure as BMad-Integrated mode}
## Recommendations
1. **High Priority (P0-P1):**
- Add E2E test for password reset flow
- Add API tests for token refresh endpoint
2. **Medium Priority (P2):**
- Add unit tests for session timeout logic
````
Ready to continue?

View File

@@ -0,0 +1,580 @@
# Automate Workflow Validation Checklist
Use this checklist to validate that the automate workflow has been executed correctly and all deliverables meet quality standards.
## Prerequisites
Before starting this workflow, verify:
- [ ] Framework scaffolding configured (playwright.config.ts or cypress.config.ts exists)
- [ ] Test directory structure exists (tests/ folder with subdirectories)
- [ ] Package.json has test framework dependencies installed
**Halt only if:** Framework scaffolding is completely missing (run `framework` workflow first)
**Note:** BMad artifacts (story, tech-spec, PRD) are OPTIONAL - workflow can run without them
---
## Step 1: Execution Mode Determination and Context Loading
### Mode Detection
- [ ] Execution mode correctly determined:
- [ ] BMad-Integrated Mode (story_file variable set) OR
- [ ] Standalone Mode (target_feature or target_files set) OR
- [ ] Auto-discover Mode (no targets specified)
### BMad Artifacts (If Available - OPTIONAL)
- [ ] Story markdown loaded (if `{story_file}` provided)
- [ ] Acceptance criteria extracted from story (if available)
- [ ] Tech-spec.md loaded (if `{use_tech_spec}` true and file exists)
- [ ] Test-design.md loaded (if `{use_test_design}` true and file exists)
- [ ] PRD.md loaded (if `{use_prd}` true and file exists)
- [ ] **Note**: Absence of BMad artifacts does NOT halt workflow
### Framework Configuration
- [ ] Test framework config loaded (playwright.config.ts or cypress.config.ts)
- [ ] Test directory structure identified from `{test_dir}`
- [ ] Existing test patterns reviewed
- [ ] Test runner capabilities noted (parallel execution, fixtures, etc.)
### Coverage Analysis
- [ ] Existing test files searched in `{test_dir}` (if `{analyze_coverage}` true)
- [ ] Tested features vs untested features identified
- [ ] Coverage gaps mapped (tests to source files)
- [ ] Existing fixture and factory patterns checked
### Knowledge Base Fragments Loaded
- [ ] `test-levels-framework.md` - Test level selection
- [ ] `test-priorities.md` - Priority classification (P0-P3)
- [ ] `fixture-architecture.md` - Fixture patterns with auto-cleanup
- [ ] `data-factories.md` - Factory patterns using faker
- [ ] `selective-testing.md` - Targeted test execution strategies
- [ ] `ci-burn-in.md` - Flaky test detection patterns
- [ ] `test-quality.md` - Test design principles
---
## Step 2: Automation Targets Identification
### Target Determination
**BMad-Integrated Mode (if story available):**
- [ ] Acceptance criteria mapped to test scenarios
- [ ] Features implemented in story identified
- [ ] Existing ATDD tests checked (if any)
- [ ] Expansion beyond ATDD planned (edge cases, negative paths)
**Standalone Mode (if no story):**
- [ ] Specific feature analyzed (if `{target_feature}` specified)
- [ ] Specific files analyzed (if `{target_files}` specified)
- [ ] Features auto-discovered (if `{auto_discover_features}` true)
- [ ] Features prioritized by:
- [ ] No test coverage (highest priority)
- [ ] Complex business logic
- [ ] External integrations (API, database, auth)
- [ ] Critical user paths (login, checkout, etc.)
### Test Level Selection
- [ ] Test level selection framework applied (from `test-levels-framework.md`)
- [ ] E2E tests identified: Critical user journeys, multi-system integration
- [ ] API tests identified: Business logic, service contracts, data transformations
- [ ] Component tests identified: UI behavior, interactions, state management
- [ ] Unit tests identified: Pure logic, edge cases, error handling
### Duplicate Coverage Avoidance
- [ ] Same behavior NOT tested at multiple levels unnecessarily
- [ ] E2E used for critical happy path only
- [ ] API tests used for business logic variations
- [ ] Component tests used for UI interaction edge cases
- [ ] Unit tests used for pure logic edge cases
### Priority Assignment
- [ ] Test priorities assigned using `test-priorities.md` framework
- [ ] P0 tests: Critical paths, security-critical, data integrity
- [ ] P1 tests: Important features, integration points, error handling
- [ ] P2 tests: Edge cases, less-critical variations, performance
- [ ] P3 tests: Nice-to-have, rarely-used features, exploratory
- [ ] Priority variables respected:
- [ ] `{include_p0}` = true (always include)
- [ ] `{include_p1}` = true (high priority)
- [ ] `{include_p2}` = true (medium priority)
- [ ] `{include_p3}` = false (low priority, skip by default)
### Coverage Plan Created
- [ ] Test coverage plan documented
- [ ] What will be tested at each level listed
- [ ] Priorities assigned to each test
- [ ] Coverage strategy clear (critical-paths, comprehensive, or selective)
---
## Step 3: Test Infrastructure Generated
### Fixture Architecture
- [ ] Existing fixtures checked in `tests/support/fixtures/`
- [ ] Fixture architecture created/enhanced (if `{generate_fixtures}` true)
- [ ] All fixtures use Playwright's `test.extend()` pattern
- [ ] All fixtures have auto-cleanup in teardown
- [ ] Common fixtures created/enhanced:
- [ ] authenticatedUser (with auto-delete)
- [ ] apiRequest (authenticated client)
- [ ] mockNetwork (external service mocking)
- [ ] testDatabase (with auto-cleanup)
### Data Factories
- [ ] Existing factories checked in `tests/support/factories/`
- [ ] Factory architecture created/enhanced (if `{generate_factories}` true)
- [ ] All factories use `@faker-js/faker` for random data (no hardcoded values)
- [ ] All factories support overrides for specific scenarios
- [ ] Common factories created/enhanced:
- [ ] User factory (email, password, name, role)
- [ ] Product factory (name, price, SKU)
- [ ] Order factory (items, total, status)
- [ ] Cleanup helpers provided (e.g., deleteUser(), deleteProduct())
### Helper Utilities
- [ ] Existing helpers checked in `tests/support/helpers/` (if `{update_helpers}` true)
- [ ] Common utilities created/enhanced:
- [ ] waitFor (polling for complex conditions)
- [ ] retry (retry helper for flaky operations)
- [ ] testData (test data generation)
- [ ] assertions (custom assertion helpers)
---
## Step 4: Test Files Generated
### Test File Structure
- [ ] Test files organized correctly:
- [ ] `tests/e2e/` for E2E tests
- [ ] `tests/api/` for API tests
- [ ] `tests/component/` for component tests
- [ ] `tests/unit/` for unit tests
- [ ] `tests/support/` for fixtures/factories/helpers
### E2E Tests (If Applicable)
- [ ] E2E test files created in `tests/e2e/`
- [ ] All tests follow Given-When-Then format
- [ ] All tests have priority tags ([P0], [P1], [P2], [P3]) in test name
- [ ] All tests use data-testid selectors (not CSS classes)
- [ ] One assertion per test (atomic design)
- [ ] No hard waits or sleeps (explicit waits only)
- [ ] Network-first pattern applied (route interception BEFORE navigation)
- [ ] Clear Given-When-Then comments in test code
### API Tests (If Applicable)
- [ ] API test files created in `tests/api/`
- [ ] All tests follow Given-When-Then format
- [ ] All tests have priority tags in test name
- [ ] API contracts validated (request/response structure)
- [ ] HTTP status codes verified
- [ ] Response body validation includes required fields
- [ ] Error cases tested (400, 401, 403, 404, 500)
- [ ] JWT token format validated (if auth tests)
### Component Tests (If Applicable)
- [ ] Component test files created in `tests/component/`
- [ ] All tests follow Given-When-Then format
- [ ] All tests have priority tags in test name
- [ ] Component mounting works correctly
- [ ] Interaction testing covers user actions (click, hover, keyboard)
- [ ] State management validated
- [ ] Props and events tested
### Unit Tests (If Applicable)
- [ ] Unit test files created in `tests/unit/`
- [ ] All tests follow Given-When-Then format
- [ ] All tests have priority tags in test name
- [ ] Pure logic tested (no dependencies)
- [ ] Edge cases covered
- [ ] Error handling tested
### Quality Standards Enforced
- [ ] All tests use Given-When-Then format with clear comments
- [ ] All tests have descriptive names with priority tags
- [ ] No duplicate tests (same behavior tested multiple times)
- [ ] No flaky patterns (race conditions, timing issues)
- [ ] No test interdependencies (tests can run in any order)
- [ ] Tests are deterministic (same input always produces same result)
- [ ] All tests use data-testid selectors (E2E tests)
- [ ] No hard waits: `await page.waitForTimeout()` (forbidden)
- [ ] No conditional flow: `if (await element.isVisible())` (forbidden)
- [ ] No try-catch for test logic (only for cleanup)
- [ ] No hardcoded test data (use factories with faker)
- [ ] No page object classes (tests are direct and simple)
- [ ] No shared state between tests
### Network-First Pattern Applied
- [ ] Route interception set up BEFORE navigation (E2E tests with network requests)
- [ ] `page.route()` called before `page.goto()` to prevent race conditions
- [ ] Network-first pattern verified in all E2E tests that make API calls
---
## Step 5: Test Validation and Healing (NEW - Phase 2.5)
### Healing Configuration
- [ ] Healing configuration checked:
- [ ] `{auto_validate}` setting noted (default: true)
- [ ] `{auto_heal_failures}` setting noted (default: false)
- [ ] `{max_healing_iterations}` setting noted (default: 3)
- [ ] `{use_mcp_healing}` setting noted (default: true)
### Healing Knowledge Fragments Loaded (If Healing Enabled)
- [ ] `test-healing-patterns.md` loaded (common failure patterns and fixes)
- [ ] `selector-resilience.md` loaded (selector refactoring guide)
- [ ] `timing-debugging.md` loaded (race condition fixes)
### Test Execution and Validation
- [ ] Generated tests executed (if `{auto_validate}` true)
- [ ] Test results captured:
- [ ] Total tests run
- [ ] Passing tests count
- [ ] Failing tests count
- [ ] Error messages and stack traces captured
### Healing Loop (If Enabled and Tests Failed)
- [ ] Healing loop entered (if `{auto_heal_failures}` true AND tests failed)
- [ ] For each failing test:
- [ ] Failure pattern identified (selector, timing, data, network, hard wait)
- [ ] Appropriate healing strategy applied:
- [ ] Stale selector → Replaced with data-testid or ARIA role
- [ ] Race condition → Added network-first interception or state waits
- [ ] Dynamic data → Replaced hardcoded values with regex/dynamic generation
- [ ] Network error → Added route mocking
- [ ] Hard wait → Replaced with event-based wait
- [ ] Healed test re-run to validate fix
- [ ] Iteration count tracked (max 3 attempts)
### Unfixable Tests Handling
- [ ] Tests that couldn't be healed after 3 iterations marked with `test.fixme()` (if `{mark_unhealable_as_fixme}` true)
- [ ] Detailed comment added to test.fixme() tests:
- [ ] What failure occurred
- [ ] What healing was attempted (3 iterations)
- [ ] Why healing failed
- [ ] Manual investigation steps needed
- [ ] Original test logic preserved in comments
### Healing Report Generated
- [ ] Healing report generated (if healing attempted)
- [ ] Report includes:
- [ ] Auto-heal enabled status
- [ ] Healing mode (MCP-assisted or Pattern-based)
- [ ] Iterations allowed (max_healing_iterations)
- [ ] Validation results (total, passing, failing)
- [ ] Successfully healed tests (count, file:line, fix applied)
- [ ] Unable to heal tests (count, file:line, reason)
- [ ] Healing patterns applied (selector fixes, timing fixes, data fixes)
- [ ] Knowledge base references used
---
## Step 6: Documentation and Scripts Updated
### Test README Updated
- [ ] `tests/README.md` created or updated (if `{update_readme}` true)
- [ ] Test suite structure overview included
- [ ] Test execution instructions provided (all, specific files, by priority)
- [ ] Fixture usage examples provided
- [ ] Factory usage examples provided
- [ ] Priority tagging convention explained ([P0], [P1], [P2], [P3])
- [ ] How to write new tests documented
- [ ] Common patterns documented
- [ ] Anti-patterns documented (what to avoid)
### package.json Scripts Updated
- [ ] package.json scripts added/updated (if `{update_package_scripts}` true)
- [ ] `test:e2e` script for all E2E tests
- [ ] `test:e2e:p0` script for P0 tests only
- [ ] `test:e2e:p1` script for P0 + P1 tests
- [ ] `test:api` script for API tests
- [ ] `test:component` script for component tests
- [ ] `test:unit` script for unit tests (if applicable)
### Test Suite Executed
- [ ] Test suite run locally (if `{run_tests_after_generation}` true)
- [ ] Test results captured (passing/failing counts)
- [ ] No flaky patterns detected (tests are deterministic)
- [ ] Setup requirements documented (if any)
- [ ] Known issues documented (if any)
---
## Step 6: Automation Summary Generated
### Automation Summary Document
- [ ] Output file created at `{output_summary}`
- [ ] Document includes execution mode (BMad-Integrated, Standalone, Auto-discover)
- [ ] Feature analysis included (source files, coverage gaps) - Standalone mode
- [ ] Tests created listed (E2E, API, Component, Unit) with counts and paths
- [ ] Infrastructure created listed (fixtures, factories, helpers)
- [ ] Test execution instructions provided
- [ ] Coverage analysis included:
- [ ] Total test count
- [ ] Priority breakdown (P0, P1, P2, P3 counts)
- [ ] Test level breakdown (E2E, API, Component, Unit counts)
- [ ] Coverage percentage (if calculated)
- [ ] Coverage status (acceptance criteria covered, gaps identified)
- [ ] Definition of Done checklist included
- [ ] Next steps provided
- [ ] Recommendations included (if Standalone mode)
### Summary Provided to User
- [ ] Concise summary output provided
- [ ] Total tests created across test levels
- [ ] Priority breakdown (P0, P1, P2, P3 counts)
- [ ] Infrastructure counts (fixtures, factories, helpers)
- [ ] Test execution command provided
- [ ] Output file path provided
- [ ] Next steps listed
---
## Quality Checks
### Test Design Quality
- [ ] Tests are readable (clear Given-When-Then structure)
- [ ] Tests are maintainable (use factories/fixtures, not hardcoded data)
- [ ] Tests are isolated (no shared state between tests)
- [ ] Tests are deterministic (no race conditions or flaky patterns)
- [ ] Tests are atomic (one assertion per test)
- [ ] Tests are fast (no unnecessary waits or delays)
- [ ] Tests are lean (files under {max_file_lines} lines)
### Knowledge Base Integration
- [ ] Test level selection framework applied (from `test-levels-framework.md`)
- [ ] Priority classification applied (from `test-priorities.md`)
- [ ] Fixture architecture patterns applied (from `fixture-architecture.md`)
- [ ] Data factory patterns applied (from `data-factories.md`)
- [ ] Selective testing strategies considered (from `selective-testing.md`)
- [ ] Flaky test detection patterns considered (from `ci-burn-in.md`)
- [ ] Test quality principles applied (from `test-quality.md`)
### Code Quality
- [ ] All TypeScript types are correct and complete
- [ ] No linting errors in generated test files
- [ ] Consistent naming conventions followed
- [ ] Imports are organized and correct
- [ ] Code follows project style guide
- [ ] No console.log or debug statements in test code
---
## Integration Points
### With Framework Workflow
- [ ] Test framework configuration detected and used
- [ ] Directory structure matches framework setup
- [ ] Fixtures and helpers follow established patterns
- [ ] Naming conventions consistent with framework standards
### With BMad Workflows (If Available - OPTIONAL)
**With Story Workflow:**
- [ ] Story ID correctly referenced in output (if story available)
- [ ] Acceptance criteria from story reflected in tests (if story available)
- [ ] Technical constraints from story considered (if story available)
**With test-design Workflow:**
- [ ] P0 scenarios from test-design prioritized (if test-design available)
- [ ] Risk assessment from test-design considered (if test-design available)
- [ ] Coverage strategy aligned with test-design (if test-design available)
**With atdd Workflow:**
- [ ] Existing ATDD tests checked (if story had ATDD workflow run)
- [ ] Expansion beyond ATDD planned (edge cases, negative paths)
- [ ] No duplicate coverage with ATDD tests
### With CI Pipeline
- [ ] Tests can run in CI environment
- [ ] Tests are parallelizable (no shared state)
- [ ] Tests have appropriate timeouts
- [ ] Tests clean up their data (no CI environment pollution)
---
## Completion Criteria
All of the following must be true before marking this workflow as complete:
- [ ] **Execution mode determined** (BMad-Integrated, Standalone, or Auto-discover)
- [ ] **Framework configuration loaded** and validated
- [ ] **Coverage analysis completed** (gaps identified if analyze_coverage true)
- [ ] **Automation targets identified** (what needs testing)
- [ ] **Test levels selected** appropriately (E2E, API, Component, Unit)
- [ ] **Duplicate coverage avoided** (same behavior not tested at multiple levels)
- [ ] **Test priorities assigned** (P0, P1, P2, P3)
- [ ] **Fixture architecture created/enhanced** with auto-cleanup
- [ ] **Data factories created/enhanced** using faker (no hardcoded data)
- [ ] **Helper utilities created/enhanced** (if needed)
- [ ] **Test files generated** at appropriate levels (E2E, API, Component, Unit)
- [ ] **Given-When-Then format used** consistently across all tests
- [ ] **Priority tags added** to all test names ([P0], [P1], [P2], [P3])
- [ ] **data-testid selectors used** in E2E tests (not CSS classes)
- [ ] **Network-first pattern applied** (route interception before navigation)
- [ ] **Quality standards enforced** (no hard waits, no flaky patterns, self-cleaning, deterministic)
- [ ] **Test README updated** with execution instructions and patterns
- [ ] **package.json scripts updated** with test execution commands
- [ ] **Test suite run locally** (if run_tests_after_generation true)
- [ ] **Tests validated** (if auto_validate enabled)
- [ ] **Failures healed** (if auto_heal_failures enabled and tests failed)
- [ ] **Healing report generated** (if healing attempted)
- [ ] **Unfixable tests marked** with test.fixme() and detailed comments (if any)
- [ ] **Automation summary created** and saved to correct location
- [ ] **Output file formatted correctly**
- [ ] **Knowledge base references applied** and documented (including healing fragments if used)
- [ ] **No test quality issues** (flaky patterns, race conditions, hardcoded data, page objects)
---
## Common Issues and Resolutions
### Issue: BMad artifacts not found
**Problem:** Story, tech-spec, or PRD files not found when variables are set.
**Resolution:**
- **automate does NOT require BMad artifacts** - they are OPTIONAL enhancements
- If files not found, switch to Standalone Mode automatically
- Analyze source code directly without BMad context
- Continue workflow without halting
### Issue: Framework configuration not found
**Problem:** No playwright.config.ts or cypress.config.ts found.
**Resolution:**
- **HALT workflow** - framework is required
- Message: "Framework scaffolding required. Run `bmad tea *framework` first."
- User must run framework workflow before automate
### Issue: No automation targets identified
**Problem:** Neither story, target_feature, nor target_files specified, and auto-discover finds nothing.
**Resolution:**
- Check if source_dir variable is correct
- Verify source code exists in project
- Ask user to specify target_feature or target_files explicitly
- Provide examples: `target_feature: "src/auth/"` or `target_files: "src/auth/login.ts,src/auth/session.ts"`
### Issue: Duplicate coverage detected
**Problem:** Same behavior tested at multiple levels (E2E + API + Component).
**Resolution:**
- Review test level selection framework (test-levels-framework.md)
- Use E2E for critical happy path ONLY
- Use API for business logic variations
- Use Component for UI edge cases
- Remove redundant tests that duplicate coverage
### Issue: Tests have hardcoded data
**Problem:** Tests use hardcoded email addresses, passwords, or other data.
**Resolution:**
- Replace all hardcoded data with factory function calls
- Use faker for all random data generation
- Update data-factories to support all required test scenarios
- Example: `createUser({ email: faker.internet.email() })`
### Issue: Tests are flaky
**Problem:** Tests fail intermittently, pass on retry.
**Resolution:**
- Remove all hard waits (`page.waitForTimeout()`)
- Use explicit waits (`page.waitForSelector()`)
- Apply network-first pattern (route interception before navigation)
- Remove conditional flow (`if (await element.isVisible())`)
- Ensure tests are deterministic (no race conditions)
- Run burn-in loop (10 iterations) to detect flakiness
### Issue: Fixtures don't clean up data
**Problem:** Test data persists after test run, causing test pollution.
**Resolution:**
- Ensure all fixtures have cleanup in teardown phase
- Cleanup happens AFTER `await use(data)`
- Call deletion/cleanup functions (deleteUser, deleteProduct, etc.)
- Verify cleanup works by checking database/storage after test run
### Issue: Tests too slow
**Problem:** Tests take longer than 90 seconds (max_test_duration).
**Resolution:**
- Remove unnecessary waits and delays
- Use parallel execution where possible
- Mock external services (don't make real API calls)
- Use API tests instead of E2E for business logic
- Optimize test data creation (use in-memory database, etc.)
---
## Notes for TEA Agent
- **automate is flexible:** Can work with or without BMad artifacts (story, tech-spec, PRD are OPTIONAL)
- **Standalone mode is powerful:** Analyze any codebase and generate tests independently
- **Auto-discover mode:** Scan codebase for features needing tests when no targets specified
- **Framework is the ONLY hard requirement:** HALT if framework config missing, otherwise proceed
- **Avoid duplicate coverage:** E2E for critical paths only, API/Component for variations
- **Priority tagging enables selective execution:** P0 tests run on every commit, P1 on PR, P2 nightly
- **Network-first pattern prevents race conditions:** Route interception BEFORE navigation
- **No page objects:** Keep tests simple, direct, and maintainable
- **Use knowledge base:** Load relevant fragments (test-levels, test-priorities, fixture-architecture, data-factories, healing patterns) for guidance
- **Deterministic tests only:** No hard waits, no conditional flow, no flaky patterns allowed
- **Optional healing:** auto_heal_failures disabled by default (opt-in for automatic test healing)
- **Graceful degradation:** Healing works without Playwright MCP (pattern-based fallback)
- **Unfixable tests handled:** Mark with test.fixme() and detailed comments (not silently broken)

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,61 @@
# Test Architect workflow: automate
name: testarch-automate
description: "Expand test automation coverage after implementation or analyze existing codebase to generate comprehensive test suite"
author: "BMad"
# Critical variables from config
config_source: "{project-root}/bmad/bmm/config.yaml"
output_folder: "{config_source}:output_folder"
user_name: "{config_source}:user_name"
communication_language: "{config_source}:communication_language"
document_output_language: "{config_source}:document_output_language"
date: system-generated
# Workflow components
installed_path: "{project-root}/bmad/bmm/workflows/testarch/automate"
instructions: "{installed_path}/instructions.md"
validation: "{installed_path}/checklist.md"
template: false
# Variables and inputs
variables:
# Execution mode and targeting
standalone_mode: true # Can work without BMad artifacts (true) or integrate with BMad (false)
coverage_target: "critical-paths" # critical-paths, comprehensive, selective
# Directory paths
test_dir: "{project-root}/tests" # Root test directory
source_dir: "{project-root}/src" # Source code directory
# Output configuration
default_output_file: "{output_folder}/automation-summary.md"
# Required tools
required_tools:
- read_file # Read source code, existing tests, BMad artifacts
- write_file # Create test files, fixtures, factories, summaries
- create_directory # Create test directories
- list_files # Discover features and existing tests
- search_repo # Find coverage gaps and patterns
- glob # Find test files and source files
# Recommended inputs (optional - depends on mode)
recommended_inputs:
- story: "Story markdown with acceptance criteria (optional - BMad mode only)"
- tech_spec: "Technical specification (optional - BMad mode only)"
- test_design: "Test design document with risk/priority (optional - BMad mode only)"
- source_code: "Feature implementation to analyze (required for standalone mode)"
- existing_tests: "Current test suite for gap analysis (always helpful)"
- framework_config: "Test framework configuration (playwright.config.ts, cypress.config.ts)"
tags:
- qa
- automation
- test-architect
- regression
- coverage
execution_hints:
interactive: false # Minimize prompts
autonomous: true # Proceed without user input unless blocked
iterative: true