arrr-erp-test-v2/README.md

Full ERP UAT + VAPT Suite v2.3

This is a separate Playwright testing project for Audit Firm ERP. It can run from your local system or from a separate Coolify/Node container against the ERP URL.

What v2.3 includes

• Updated checklist workbook with v2.0.4 security/FY additions.
• 272 source use cases in data/checklist-rows.json.
• 1,296 generated runnable/traceable variants in data/generated-test-matrix.json.
• Existing v2.2 coverage for UAT, RBAC, validation, boundary, CSRF, IDOR, upload and security header checks.
• New targeted v2.0.4 checks for:
  • Forgot-password reset token exposure
  • Reset-token rejection
  • API login brute-force safety
  • X-Tenant-Code, X-Branch-Code, X-Year-Code spoofing lockdown
  • Trusted context-header secret mode
  • Active FY selector/session visibility
  • Active FY transactional screen smoke test
  • Locked FY write protection
  • Backup export access protection
  • Executable/oversized upload rejection
  • Storage-agent authentication
  • Cookie/security header checks
  • Optional strict CSP check
• Seed data helper: seed/seed_data.py.
• OTP auto-read support when this suite starts ERP itself.
• Excel result update support.

Important truth

This suite gives strong repeatable UAT/VAPT regression coverage, but it does not replace a professional manual VAPT. Some tests are intentionally defensive and skip when the corresponding seed IDs or optional production flags are not configured.

For strongest results:

1. Run ERP migrations.
2. Run seed/seed_data.py.
3. Paste emitted IDs into .env.
4. Run targeted v2.0.4 checks using npm run test:v204.
5. Run the full matrix using npm run test:excel.

Folder layout

playwright_full_erp_uat_vapt_v22
├─ Audit_Firm_ERP_Master_UAT_VAPT_Checklist.xlsx
├─ data/
│  ├─ checklist-rows.json
│  ├─ generated-test-matrix.json
│  └─ summary.json
├─ fixtures/
│  ├─ auth.js
│  ├─ assertions.js
│  └─ v204-helpers.js
├─ scripts/
├─ seed/
│  └─ seed_data.py
├─ tests/
│  ├─ full-excel-variants.spec.js
│  ├─ vapt-targeted.spec.js
│  └─ v204-security-additions.spec.js
└─ test-data/uploads/

Step 1 — Install

npm install
npx playwright install chromium
copy .env.example .env

Step 2 — Seed ERP data

From your ERP project root, after migrations:

python D:\path\to\playwright_full_erp_uat_vapt_v22\seed\seed_data.py

The script prints IDs like:

TENANT_A_ID=...
TENANT_B_ID=...
CLIENT_A_ID=...
NOTICE_CASE_A_ID=...

Paste those into .env of this Playwright suite.

Step 3 — Configure .env

Minimum:

BASE_URL=http://127.0.0.1:8000
SYSTEM_ADMIN_EMAIL=admin@auditfirm.local
SYSTEM_ADMIN_PASSWORD=your-password
FIRM_ADMIN_EMAIL=uat.firmadmin@tenant-a.test
FIRM_ADMIN_PASSWORD=Password@123
STAFF_EMAIL=uat.staff@tenant-a.test
STAFF_PASSWORD=Password@123
ACTIVE_FY=2025-26
PREVIOUS_FY=2024-25
TRUST_CONTEXT_HEADERS=false

For locked-year tests, first lock a FY in ERP, then set:

LOCKED_FY=2024-25

For production cookie assertion on HTTPS:

EXPECT_SECURE_COOKIES=true

For strict CSP assertion only after inline JS/CSS is removed:

EXPECT_STRICT_CSP=true

Step 4 — Run ERP

Option A: run ERP separately:

uvicorn app.main:app --reload

Then use:

START_ERP=false
BASE_URL=http://127.0.0.1:8000
STATIC_OTP=123456

Option B: let Playwright start ERP and read OTP from console:

START_ERP=true
OTP_FROM_CONSOLE=true
ERP_WORKDIR=D:\01 Project\versions\version 2.0\audit_firm_v2_baseline_with_alembic\Audit_Firm_v2.0.3.6\Audit_Firm_v2.0.3.6
ERP_COMMAND=uvicorn app.main:app --reload

Step 5 — Check matrix and seed config

npm run matrix
npm run test:seed-check

Expected v2.3 summary:

Source Excel rows: 272
Generated variants: 1296
v204AdditionalChecks: 16

Step 6 — Run tests

Full Excel mapped suite:

npm run test:excel

Original targeted VAPT checks:

npm run test:vapt

New v2.0.4 targeted hardening checks:

npm run test:v204

All tests:

npm run test:all

Show report:

npm run show-report

Step 7 — Update Excel result status

npm run report:json
npm run update:excel

Output:

results/Audit_Firm_ERP_Master_UAT_VAPT_Checklist_v2_2_Results.xlsx

Coolify test-runner usage

You can run this suite as a separate Node/Playwright container in Coolify. Set:

BASE_URL=https://uat-erp.yourdomain.com
START_ERP=false
TRUST_CONTEXT_HEADERS=false

Do not run heavy load tests from the same VPS that hosts the ERP. Use this suite for functional UAT, regression, and security flow checks.

Recommended ERP UI improvement for stronger automation

Add stable selectors over time:

<select data-testid="active-fy-select"></select>
<button data-testid="client-create-submit">Save</button>
<input data-testid="client-pan">
<a data-testid="notice-case-new">New Case</a>

Without stable selectors, Playwright uses defensive generic selectors, which is useful but less precise.

Security note

Keep public deployment with:

TRUST_CONTEXT_HEADERS=false

Trusted internal context headers should only be used for backend-to-backend automation and must include X-AuditFirm-Context-Secret when enabled.

---

v2.4.0 changelog

• Checklist reconciled to automation. The Excel checklist is now regenerated from the Playwright source matrix (data/checklist-rows.json, 272 source cases / 1296 variants) so every automated case has exactly one tracking row. New file: Audit_Firm_ERP_Master_UAT_VAPT_Checklist_v2.4.xlsx (302 rows total).
• Added UAT_Work, UAT_Domains, VAPT_Domains, V204_Additional_Checks to the checklist — previously automated but missing from the tracker.
• UAT_FY_Lock_Backup (30 cases) carried in as a MANUAL block and given scaffolding in tests/fy-lock-backup.spec.js. ⚠️ This scaffolding is UNTESTED: every case is test.fixme() and must be verified against a running ERP (routes/selectors/assertions) before it counts as coverage. Run with npm run test:fy.
• 9 API-level security checks were recorded from a local SQLite run (6 Pass, 1 Partial in the v204 block, plus 2 more). All UI/browser and manual cases remain Not Started pending a full run against the deployed environment.

Known source-of-truth note

The older Audit_Firm_ERP_Master_UAT_VAPT_Checklist.xlsx (218 rows) is superseded by the v2.4 file. Use the v2.4 file going forward.