# Automate Workflow Validation Checklist Use this checklist to validate that the automate workflow has been executed correctly and all deliverables meet quality standards. ## Prerequisites Before starting this workflow, verify: - [ ] Framework scaffolding configured (playwright.config.ts or cypress.config.ts exists) - [ ] Test directory structure exists (tests/ folder with subdirectories) - [ ] Package.json has test framework dependencies installed **Halt only if:** Framework scaffolding is completely missing (run `framework` workflow first) **Note:** BMad artifacts (story, tech-spec, PRD) are OPTIONAL - workflow can run without them --- ## Step 1: Execution Mode Determination and Context Loading ### Mode Detection - [ ] Execution mode correctly determined: - [ ] BMad-Integrated Mode (story_file variable set) OR - [ ] Standalone Mode (target_feature or target_files set) OR - [ ] Auto-discover Mode (no targets specified) ### BMad Artifacts (If Available - OPTIONAL) - [ ] Story markdown loaded (if `{story_file}` provided) - [ ] Acceptance criteria extracted from story (if available) - [ ] Tech-spec.md loaded (if `{use_tech_spec}` true and file exists) - [ ] Test-design.md loaded (if `{use_test_design}` true and file exists) - [ ] PRD.md loaded (if `{use_prd}` true and file exists) - [ ] **Note**: Absence of BMad artifacts does NOT halt workflow ### Framework Configuration - [ ] Test framework config loaded (playwright.config.ts or cypress.config.ts) - [ ] Test directory structure identified from `{test_dir}` - [ ] Existing test patterns reviewed - [ ] Test runner capabilities noted (parallel execution, fixtures, etc.) ### Coverage Analysis - [ ] Existing test files searched in `{test_dir}` (if `{analyze_coverage}` true) - [ ] Tested features vs untested features identified - [ ] Coverage gaps mapped (tests to source files) - [ ] Existing fixture and factory patterns checked ### Knowledge Base Fragments Loaded - [ ] `test-levels-framework.md` - Test level selection - [ ] `test-priorities.md` - Priority classification (P0-P3) - [ ] `fixture-architecture.md` - Fixture patterns with auto-cleanup - [ ] `data-factories.md` - Factory patterns using faker - [ ] `selective-testing.md` - Targeted test execution strategies - [ ] `ci-burn-in.md` - Flaky test detection patterns - [ ] `test-quality.md` - Test design principles --- ## Step 2: Automation Targets Identification ### Target Determination **BMad-Integrated Mode (if story available):** - [ ] Acceptance criteria mapped to test scenarios - [ ] Features implemented in story identified - [ ] Existing ATDD tests checked (if any) - [ ] Expansion beyond ATDD planned (edge cases, negative paths) **Standalone Mode (if no story):** - [ ] Specific feature analyzed (if `{target_feature}` specified) - [ ] Specific files analyzed (if `{target_files}` specified) - [ ] Features auto-discovered (if `{auto_discover_features}` true) - [ ] Features prioritized by: - [ ] No test coverage (highest priority) - [ ] Complex business logic - [ ] External integrations (API, database, auth) - [ ] Critical user paths (login, checkout, etc.) ### Test Level Selection - [ ] Test level selection framework applied (from `test-levels-framework.md`) - [ ] E2E tests identified: Critical user journeys, multi-system integration - [ ] API tests identified: Business logic, service contracts, data transformations - [ ] Component tests identified: UI behavior, interactions, state management - [ ] Unit tests identified: Pure logic, edge cases, error handling ### Duplicate Coverage Avoidance - [ ] Same behavior NOT tested at multiple levels unnecessarily - [ ] E2E used for critical happy path only - [ ] API tests used for business logic variations - [ ] Component tests used for UI interaction edge cases - [ ] Unit tests used for pure logic edge cases ### Priority Assignment - [ ] Test priorities assigned using `test-priorities.md` framework - [ ] P0 tests: Critical paths, security-critical, data integrity - [ ] P1 tests: Important features, integration points, error handling - [ ] P2 tests: Edge cases, less-critical variations, performance - [ ] P3 tests: Nice-to-have, rarely-used features, exploratory - [ ] Priority variables respected: - [ ] `{include_p0}` = true (always include) - [ ] `{include_p1}` = true (high priority) - [ ] `{include_p2}` = true (medium priority) - [ ] `{include_p3}` = false (low priority, skip by default) ### Coverage Plan Created - [ ] Test coverage plan documented - [ ] What will be tested at each level listed - [ ] Priorities assigned to each test - [ ] Coverage strategy clear (critical-paths, comprehensive, or selective) --- ## Step 3: Test Infrastructure Generated ### Fixture Architecture - [ ] Existing fixtures checked in `tests/support/fixtures/` - [ ] Fixture architecture created/enhanced (if `{generate_fixtures}` true) - [ ] All fixtures use Playwright's `test.extend()` pattern - [ ] All fixtures have auto-cleanup in teardown - [ ] Common fixtures created/enhanced: - [ ] authenticatedUser (with auto-delete) - [ ] apiRequest (authenticated client) - [ ] mockNetwork (external service mocking) - [ ] testDatabase (with auto-cleanup) ### Data Factories - [ ] Existing factories checked in `tests/support/factories/` - [ ] Factory architecture created/enhanced (if `{generate_factories}` true) - [ ] All factories use `@faker-js/faker` for random data (no hardcoded values) - [ ] All factories support overrides for specific scenarios - [ ] Common factories created/enhanced: - [ ] User factory (email, password, name, role) - [ ] Product factory (name, price, SKU) - [ ] Order factory (items, total, status) - [ ] Cleanup helpers provided (e.g., deleteUser(), deleteProduct()) ### Helper Utilities - [ ] Existing helpers checked in `tests/support/helpers/` (if `{update_helpers}` true) - [ ] Common utilities created/enhanced: - [ ] waitFor (polling for complex conditions) - [ ] retry (retry helper for flaky operations) - [ ] testData (test data generation) - [ ] assertions (custom assertion helpers) --- ## Step 4: Test Files Generated ### Test File Structure - [ ] Test files organized correctly: - [ ] `tests/e2e/` for E2E tests - [ ] `tests/api/` for API tests - [ ] `tests/component/` for component tests - [ ] `tests/unit/` for unit tests - [ ] `tests/support/` for fixtures/factories/helpers ### E2E Tests (If Applicable) - [ ] E2E test files created in `tests/e2e/` - [ ] All tests follow Given-When-Then format - [ ] All tests have priority tags ([P0], [P1], [P2], [P3]) in test name - [ ] All tests use data-testid selectors (not CSS classes) - [ ] One assertion per test (atomic design) - [ ] No hard waits or sleeps (explicit waits only) - [ ] Network-first pattern applied (route interception BEFORE navigation) - [ ] Clear Given-When-Then comments in test code ### API Tests (If Applicable) - [ ] API test files created in `tests/api/` - [ ] All tests follow Given-When-Then format - [ ] All tests have priority tags in test name - [ ] API contracts validated (request/response structure) - [ ] HTTP status codes verified - [ ] Response body validation includes required fields - [ ] Error cases tested (400, 401, 403, 404, 500) - [ ] JWT token format validated (if auth tests) ### Component Tests (If Applicable) - [ ] Component test files created in `tests/component/` - [ ] All tests follow Given-When-Then format - [ ] All tests have priority tags in test name - [ ] Component mounting works correctly - [ ] Interaction testing covers user actions (click, hover, keyboard) - [ ] State management validated - [ ] Props and events tested ### Unit Tests (If Applicable) - [ ] Unit test files created in `tests/unit/` - [ ] All tests follow Given-When-Then format - [ ] All tests have priority tags in test name - [ ] Pure logic tested (no dependencies) - [ ] Edge cases covered - [ ] Error handling tested ### Quality Standards Enforced - [ ] All tests use Given-When-Then format with clear comments - [ ] All tests have descriptive names with priority tags - [ ] No duplicate tests (same behavior tested multiple times) - [ ] No flaky patterns (race conditions, timing issues) - [ ] No test interdependencies (tests can run in any order) - [ ] Tests are deterministic (same input always produces same result) - [ ] All tests use data-testid selectors (E2E tests) - [ ] No hard waits: `await page.waitForTimeout()` (forbidden) - [ ] No conditional flow: `if (await element.isVisible())` (forbidden) - [ ] No try-catch for test logic (only for cleanup) - [ ] No hardcoded test data (use factories with faker) - [ ] No page object classes (tests are direct and simple) - [ ] No shared state between tests ### Network-First Pattern Applied - [ ] Route interception set up BEFORE navigation (E2E tests with network requests) - [ ] `page.route()` called before `page.goto()` to prevent race conditions - [ ] Network-first pattern verified in all E2E tests that make API calls --- ## Step 5: Test Validation and Healing (NEW - Phase 2.5) ### Healing Configuration - [ ] Healing configuration checked: - [ ] `{auto_validate}` setting noted (default: true) - [ ] `{auto_heal_failures}` setting noted (default: false) - [ ] `{max_healing_iterations}` setting noted (default: 3) - [ ] `{use_mcp_healing}` setting noted (default: true) ### Healing Knowledge Fragments Loaded (If Healing Enabled) - [ ] `test-healing-patterns.md` loaded (common failure patterns and fixes) - [ ] `selector-resilience.md` loaded (selector refactoring guide) - [ ] `timing-debugging.md` loaded (race condition fixes) ### Test Execution and Validation - [ ] Generated tests executed (if `{auto_validate}` true) - [ ] Test results captured: - [ ] Total tests run - [ ] Passing tests count - [ ] Failing tests count - [ ] Error messages and stack traces captured ### Healing Loop (If Enabled and Tests Failed) - [ ] Healing loop entered (if `{auto_heal_failures}` true AND tests failed) - [ ] For each failing test: - [ ] Failure pattern identified (selector, timing, data, network, hard wait) - [ ] Appropriate healing strategy applied: - [ ] Stale selector → Replaced with data-testid or ARIA role - [ ] Race condition → Added network-first interception or state waits - [ ] Dynamic data → Replaced hardcoded values with regex/dynamic generation - [ ] Network error → Added route mocking - [ ] Hard wait → Replaced with event-based wait - [ ] Healed test re-run to validate fix - [ ] Iteration count tracked (max 3 attempts) ### Unfixable Tests Handling - [ ] Tests that couldn't be healed after 3 iterations marked with `test.fixme()` (if `{mark_unhealable_as_fixme}` true) - [ ] Detailed comment added to test.fixme() tests: - [ ] What failure occurred - [ ] What healing was attempted (3 iterations) - [ ] Why healing failed - [ ] Manual investigation steps needed - [ ] Original test logic preserved in comments ### Healing Report Generated - [ ] Healing report generated (if healing attempted) - [ ] Report includes: - [ ] Auto-heal enabled status - [ ] Healing mode (MCP-assisted or Pattern-based) - [ ] Iterations allowed (max_healing_iterations) - [ ] Validation results (total, passing, failing) - [ ] Successfully healed tests (count, file:line, fix applied) - [ ] Unable to heal tests (count, file:line, reason) - [ ] Healing patterns applied (selector fixes, timing fixes, data fixes) - [ ] Knowledge base references used --- ## Step 6: Documentation and Scripts Updated ### Test README Updated - [ ] `tests/README.md` created or updated (if `{update_readme}` true) - [ ] Test suite structure overview included - [ ] Test execution instructions provided (all, specific files, by priority) - [ ] Fixture usage examples provided - [ ] Factory usage examples provided - [ ] Priority tagging convention explained ([P0], [P1], [P2], [P3]) - [ ] How to write new tests documented - [ ] Common patterns documented - [ ] Anti-patterns documented (what to avoid) ### package.json Scripts Updated - [ ] package.json scripts added/updated (if `{update_package_scripts}` true) - [ ] `test:e2e` script for all E2E tests - [ ] `test:e2e:p0` script for P0 tests only - [ ] `test:e2e:p1` script for P0 + P1 tests - [ ] `test:api` script for API tests - [ ] `test:component` script for component tests - [ ] `test:unit` script for unit tests (if applicable) ### Test Suite Executed - [ ] Test suite run locally (if `{run_tests_after_generation}` true) - [ ] Test results captured (passing/failing counts) - [ ] No flaky patterns detected (tests are deterministic) - [ ] Setup requirements documented (if any) - [ ] Known issues documented (if any) --- ## Step 6: Automation Summary Generated ### Automation Summary Document - [ ] Output file created at `{output_summary}` - [ ] Document includes execution mode (BMad-Integrated, Standalone, Auto-discover) - [ ] Feature analysis included (source files, coverage gaps) - Standalone mode - [ ] Tests created listed (E2E, API, Component, Unit) with counts and paths - [ ] Infrastructure created listed (fixtures, factories, helpers) - [ ] Test execution instructions provided - [ ] Coverage analysis included: - [ ] Total test count - [ ] Priority breakdown (P0, P1, P2, P3 counts) - [ ] Test level breakdown (E2E, API, Component, Unit counts) - [ ] Coverage percentage (if calculated) - [ ] Coverage status (acceptance criteria covered, gaps identified) - [ ] Definition of Done checklist included - [ ] Next steps provided - [ ] Recommendations included (if Standalone mode) ### Summary Provided to User - [ ] Concise summary output provided - [ ] Total tests created across test levels - [ ] Priority breakdown (P0, P1, P2, P3 counts) - [ ] Infrastructure counts (fixtures, factories, helpers) - [ ] Test execution command provided - [ ] Output file path provided - [ ] Next steps listed --- ## Quality Checks ### Test Design Quality - [ ] Tests are readable (clear Given-When-Then structure) - [ ] Tests are maintainable (use factories/fixtures, not hardcoded data) - [ ] Tests are isolated (no shared state between tests) - [ ] Tests are deterministic (no race conditions or flaky patterns) - [ ] Tests are atomic (one assertion per test) - [ ] Tests are fast (no unnecessary waits or delays) - [ ] Tests are lean (files under {max_file_lines} lines) ### Knowledge Base Integration - [ ] Test level selection framework applied (from `test-levels-framework.md`) - [ ] Priority classification applied (from `test-priorities.md`) - [ ] Fixture architecture patterns applied (from `fixture-architecture.md`) - [ ] Data factory patterns applied (from `data-factories.md`) - [ ] Selective testing strategies considered (from `selective-testing.md`) - [ ] Flaky test detection patterns considered (from `ci-burn-in.md`) - [ ] Test quality principles applied (from `test-quality.md`) ### Code Quality - [ ] All TypeScript types are correct and complete - [ ] No linting errors in generated test files - [ ] Consistent naming conventions followed - [ ] Imports are organized and correct - [ ] Code follows project style guide - [ ] No console.log or debug statements in test code --- ## Integration Points ### With Framework Workflow - [ ] Test framework configuration detected and used - [ ] Directory structure matches framework setup - [ ] Fixtures and helpers follow established patterns - [ ] Naming conventions consistent with framework standards ### With BMad Workflows (If Available - OPTIONAL) **With Story Workflow:** - [ ] Story ID correctly referenced in output (if story available) - [ ] Acceptance criteria from story reflected in tests (if story available) - [ ] Technical constraints from story considered (if story available) **With test-design Workflow:** - [ ] P0 scenarios from test-design prioritized (if test-design available) - [ ] Risk assessment from test-design considered (if test-design available) - [ ] Coverage strategy aligned with test-design (if test-design available) **With atdd Workflow:** - [ ] Existing ATDD tests checked (if story had ATDD workflow run) - [ ] Expansion beyond ATDD planned (edge cases, negative paths) - [ ] No duplicate coverage with ATDD tests ### With CI Pipeline - [ ] Tests can run in CI environment - [ ] Tests are parallelizable (no shared state) - [ ] Tests have appropriate timeouts - [ ] Tests clean up their data (no CI environment pollution) --- ## Completion Criteria All of the following must be true before marking this workflow as complete: - [ ] **Execution mode determined** (BMad-Integrated, Standalone, or Auto-discover) - [ ] **Framework configuration loaded** and validated - [ ] **Coverage analysis completed** (gaps identified if analyze_coverage true) - [ ] **Automation targets identified** (what needs testing) - [ ] **Test levels selected** appropriately (E2E, API, Component, Unit) - [ ] **Duplicate coverage avoided** (same behavior not tested at multiple levels) - [ ] **Test priorities assigned** (P0, P1, P2, P3) - [ ] **Fixture architecture created/enhanced** with auto-cleanup - [ ] **Data factories created/enhanced** using faker (no hardcoded data) - [ ] **Helper utilities created/enhanced** (if needed) - [ ] **Test files generated** at appropriate levels (E2E, API, Component, Unit) - [ ] **Given-When-Then format used** consistently across all tests - [ ] **Priority tags added** to all test names ([P0], [P1], [P2], [P3]) - [ ] **data-testid selectors used** in E2E tests (not CSS classes) - [ ] **Network-first pattern applied** (route interception before navigation) - [ ] **Quality standards enforced** (no hard waits, no flaky patterns, self-cleaning, deterministic) - [ ] **Test README updated** with execution instructions and patterns - [ ] **package.json scripts updated** with test execution commands - [ ] **Test suite run locally** (if run_tests_after_generation true) - [ ] **Tests validated** (if auto_validate enabled) - [ ] **Failures healed** (if auto_heal_failures enabled and tests failed) - [ ] **Healing report generated** (if healing attempted) - [ ] **Unfixable tests marked** with test.fixme() and detailed comments (if any) - [ ] **Automation summary created** and saved to correct location - [ ] **Output file formatted correctly** - [ ] **Knowledge base references applied** and documented (including healing fragments if used) - [ ] **No test quality issues** (flaky patterns, race conditions, hardcoded data, page objects) --- ## Common Issues and Resolutions ### Issue: BMad artifacts not found **Problem:** Story, tech-spec, or PRD files not found when variables are set. **Resolution:** - **automate does NOT require BMad artifacts** - they are OPTIONAL enhancements - If files not found, switch to Standalone Mode automatically - Analyze source code directly without BMad context - Continue workflow without halting ### Issue: Framework configuration not found **Problem:** No playwright.config.ts or cypress.config.ts found. **Resolution:** - **HALT workflow** - framework is required - Message: "Framework scaffolding required. Run `bmad tea *framework` first." - User must run framework workflow before automate ### Issue: No automation targets identified **Problem:** Neither story, target_feature, nor target_files specified, and auto-discover finds nothing. **Resolution:** - Check if source_dir variable is correct - Verify source code exists in project - Ask user to specify target_feature or target_files explicitly - Provide examples: `target_feature: "src/auth/"` or `target_files: "src/auth/login.ts,src/auth/session.ts"` ### Issue: Duplicate coverage detected **Problem:** Same behavior tested at multiple levels (E2E + API + Component). **Resolution:** - Review test level selection framework (test-levels-framework.md) - Use E2E for critical happy path ONLY - Use API for business logic variations - Use Component for UI edge cases - Remove redundant tests that duplicate coverage ### Issue: Tests have hardcoded data **Problem:** Tests use hardcoded email addresses, passwords, or other data. **Resolution:** - Replace all hardcoded data with factory function calls - Use faker for all random data generation - Update data-factories to support all required test scenarios - Example: `createUser({ email: faker.internet.email() })` ### Issue: Tests are flaky **Problem:** Tests fail intermittently, pass on retry. **Resolution:** - Remove all hard waits (`page.waitForTimeout()`) - Use explicit waits (`page.waitForSelector()`) - Apply network-first pattern (route interception before navigation) - Remove conditional flow (`if (await element.isVisible())`) - Ensure tests are deterministic (no race conditions) - Run burn-in loop (10 iterations) to detect flakiness ### Issue: Fixtures don't clean up data **Problem:** Test data persists after test run, causing test pollution. **Resolution:** - Ensure all fixtures have cleanup in teardown phase - Cleanup happens AFTER `await use(data)` - Call deletion/cleanup functions (deleteUser, deleteProduct, etc.) - Verify cleanup works by checking database/storage after test run ### Issue: Tests too slow **Problem:** Tests take longer than 90 seconds (max_test_duration). **Resolution:** - Remove unnecessary waits and delays - Use parallel execution where possible - Mock external services (don't make real API calls) - Use API tests instead of E2E for business logic - Optimize test data creation (use in-memory database, etc.) --- ## Notes for TEA Agent - **automate is flexible:** Can work with or without BMad artifacts (story, tech-spec, PRD are OPTIONAL) - **Standalone mode is powerful:** Analyze any codebase and generate tests independently - **Auto-discover mode:** Scan codebase for features needing tests when no targets specified - **Framework is the ONLY hard requirement:** HALT if framework config missing, otherwise proceed - **Avoid duplicate coverage:** E2E for critical paths only, API/Component for variations - **Priority tagging enables selective execution:** P0 tests run on every commit, P1 on PR, P2 nightly - **Network-first pattern prevents race conditions:** Route interception BEFORE navigation - **No page objects:** Keep tests simple, direct, and maintainable - **Use knowledge base:** Load relevant fragments (test-levels, test-priorities, fixture-architecture, data-factories, healing patterns) for guidance - **Deterministic tests only:** No hard waits, no conditional flow, no flaky patterns allowed - **Optional healing:** auto_heal_failures disabled by default (opt-in for automatic test healing) - **Graceful degradation:** Healing works without Playwright MCP (pattern-based fallback) - **Unfixable tests handled:** Mark with test.fixme() and detailed comments (not silently broken)