Evidence and documentation validation
Manual home · Coverage · Gaps
What was reviewed
Baseline: 57cfc40423f60ac87e66988fb8e82169c5de7d90. 783 fetched source/document/test files form the local inspection corpus. Sources included all TypeScript API implementation files, tenant feature implementation, direct shared contract files, application routing/navigation, the internal Python runtime, existing architecture/runbook documents and relevant tests. No fetched project reference file under sources/ was edited. The source manifest below lists paths and Git blob SHAs for reproducibility, without embedding code or customer data.
The coverage extractor parsed API TypeScript/TSX with zero parser errors, matched router factories to composition mounts and expanded finite generic resource/action/phase selectors. There were zero unresolved router mounts and zero unmapped resulting entries. Public health, service callbacks, provider/OAuth endpoints and platform operations were included, even where they are supporting processes rather than end-user screens.
Related test source inspected
The examples below are assertion titles present in existing suites. They support understanding of implementation intent and edge cases; they are not test execution results, and a related suite does not prove every step or role in a workflow has coverage. Missing end-to-end/runtime evidence remains GAP-05.
| Domain | Related suite | Example assertion titles from source |
|---|---|---|
| Access and setup | services/api/tests/local-auth.test.ts | issues a tenant-scoped token and applies role permissions; rejects invalid credentials and missing bearer tokens |
| Access and setup | services/api/tests/authenticator.test.ts | fails closed outside development when no provider is configured; allows development without a provider and any configured provider |
| Access and setup | services/api/tests/admin-members.test.ts | does not let a BPO tenant assign a client company from another tenant; lets an owner invite, promote, suspend and deactivate members |
| Access and setup | services/api/tests/tenant-settings.test.ts | keeps journey coaching disabled by default and lets only an owner enable the tenant gate; lets an owner set the workspace timezone |
| Interaction processing | services/api/tests/interaction-imports.test.ts | saves profiles and imports/replays normalized conversations; requires retention and reports individual import conflicts without rolling back successes |
| Interaction processing | services/api/tests/interaction-deletion.test.ts | deletes the analysis and AI suggestions with the interaction; leaves derived records for other tenants untouched |
| Interaction processing | services/api/tests/interaction-batch.test.ts | archives and restores interactions while filtering active records by default; returns tenant-scoped interactions in the requested order, skipping unknowns |
| Interaction processing | services/api/tests/conversations.test.ts | ingests live chat idempotently, stores the conversation and projects turns; ingests email threads and requires a subject |
| Interaction processing | services/api/tests/transcription-jobs.test.ts | lists active jobs joined with the tenant; finds a tenant job pushed beyond a single upstream page |
| Interaction processing | services/api/tests/transcription-controls.test.ts | loads legacy transcription settings with an empty profile list and preserves the glossary; snapshots settings at upload and safely retries an uncertain enqueue |
| Interaction processing | services/api/tests/transcription-results.test.ts | ingests idempotently and exposes the tenant-scoped result; rejects invalid keys, malformed payloads, and conflicting retries |
| Interaction processing | services/api/tests/upload-authorization.test.ts | returns short-lived instructions with signed upload metadata; verifies authoritative object length and SHA-256 metadata |
| Interaction processing | services/api/tests/reanalysis.test.ts | enqueues stale analyses through the durable job queue; is idempotent and skips current analyses |
| Evaluation configuration | services/api/tests/atom-library.test.ts | searches and filters the published library with stable cursor pages; exposes bounded Atom library search, filters, and cursors through the API |
| Evaluation configuration | services/api/tests/molecules.test.ts | searches and filters Molecules across stable tenant-scoped cursor pages; creates, publishes, revises and retires immutable versions |
| Evaluation configuration | services/api/tests/scorecard-suggestions.test.ts | marks provider failures as failed instead of completing with placeholders; uses all available transcript-ready interactions below the 100 interaction cap |
| Evaluation configuration | services/api/tests/scorecard-suggestion-generator.test.ts | composes three schema-valid candidates from the published molecule library; accepts JSON wrapped in a markdown fence from the model |
| Quality management | services/api/tests/app.test.ts | reports liveness without tenant context; requires a valid tenant for QMS data |
| Quality management | services/api/tests/audit-scoring.test.ts | previews the weighted score from AI-resolved sections and excludes unresolved ones; applies molecule weights and critical-failure overrides |
| Quality management | services/api/tests/audit-knowledge-flow.test.ts | requires explicit no-match confirmation and submits an all-unscored audit without a numeric result; accepts a valid pinned article section citation when the reviewer selects a possible match |
| Quality management | services/api/tests/ai-suggestions.test.ts | keeps tenant isolation and makes failed sets retryable; tracks queued batches and completes idempotently |
| Quality management | services/api/tests/ai-suggestion-policies.test.ts | rejects evidence for a no-quote atom even when the payload omits the policy; accepts an evidence-free suggestion for a no-quote atom |
| Quality management | services/api/tests/sampling-selection.test.ts | never selects when no active policy matches; always selects when a matching policy samples at 100% |
| Knowledge | services/api/tests/knowledge-cms.test.ts | manages tenant-scoped taxonomies and propagates renames; applies reviewed enrichment to a published article and retains a revision |
| Knowledge | services/api/tests/knowledge-attachments.test.ts | authorizes, completes, serves, and protects an inline image asset; keeps a published non-inline attachment during cleanup |
| Knowledge | services/api/tests/knowledge-announcements.test.ts | snapshots active agents, counts reads per send, and deduplicates opens; restricts delivery to a team audience and records partial email failures |
| Knowledge | services/api/tests/knowledge-retrieval.test.ts | continues lexical search past the first published page; searches published tenant content, cites sections, follows related links, and enforces budgets |
| Coaching and learning | services/api/tests/coaching-plans.test.ts | captures a 30-day baseline, scopes access, records mutations, and notifies the assigned agent; uses the final draft source agent when activation captures a baseline |
| Coaching and learning | services/api/tests/training.test.ts | publishes immutable revisions, assigns a team snapshot, and tracks learner completion without exposing answer keys; rejects publishing an empty draft and blocks learner mutations by other users |
| Journeys and improvement | services/api/tests/journey-config.test.ts | enriches each journey item type and meters the provider calls; rejects invalid LLM output and records the failed call |
| Journeys and improvement | services/api/tests/candidate-reviews.test.ts | finds candidate records only within the same tenant and source analysis; persists one candidate per callback and is idempotent on replay |
| Journeys and improvement | services/api/tests/journeys.test.ts | shows ready unlinked interactions as standalone journeys and reports readiness and detected episode time; uses canonical action count as the action-success sample denominator |
| Journeys and improvement | services/api/tests/journey-e2e.test.ts | ingests voice, chat and email, completes analysis and calculates one correlated journey |
| Journeys and improvement | services/api/tests/workspaces.test.ts | persists and reads tenant-scoped preferences for the authenticated user; isolates preferences across tenants |
| Insights and work | services/api/tests/command-centre.test.ts | computes audit-derived metrics from the mutation trail; honours the requested reporting window |
| Insights and work | services/api/tests/reports-search.test.ts | OpenAPI documents critical failures in the required report summary; summarizes all residual-filter matches and returns filter options beyond 1,000 audits |
| Insights and work | services/api/tests/pilot-analytics.test.ts | reports period coverage, calibration agreement, and distinct reviewer adoption without exposing identities; keeps events tenant-scoped and rejects invalid ranges or missing QMS permission |
| Insights and work | services/api/tests/alerts.test.ts | suppresses named rankings until five submitted reviews exist; groups performance reviews by linked agent account and carries its display name |
| Insights and work | services/api/tests/alert-rules.test.ts | fires a score rule only when the comparator breaches the threshold; honours the team and channel scope filters |
| Insights and work | services/api/tests/work-notifications.test.ts | counts each open audit, calibration, and re-review once and treats active aging alerts as overdue; paginates within the recipient tenant and marks individual and all notifications read |
| Insights and work | services/api/tests/mutation-trail.test.ts | appends bounded actor metadata in the same unit of work; does not append an event for a replay or rejected mutation result |
| AI assistance | services/api/tests/ask-sentinel.test.ts | keeps quality feedback separate from private conversations and shares only redacted opt-in excerpts; keeps saved conversations and jobs owned by the user without putting prompts in jobs |
| AI assistance | services/api/tests/ask-sentinel-curated-quality.test.ts | accepts the expected answer only with an accessible tenant citation; fails closed when the cited article is no longer accessible |
| AI assistance | services/api/tests/ask-sentinel-platform-quality.test.ts | requires a platform session, exposes only shared excerpts, and records the platform reviewer |
| AI assistance | services/api/tests/ai-learning.test.ts | keeps drafts inactive, scopes records by tenant, and activates versioned rollback drafts; turns only reviewed Ask Sentinel evaluations into quality signals |
| AI assistance | services/api/tests/ai-evaluation.test.ts | requires reviewed feedback for imported cases and runs tenant cases against the selected model; promotes a qualifying run only for the requested tenant and restores the previous release on rollback |
| AI assistance | services/api/tests/live-assist.test.ts | keeps ordered redacted events private and cites only active tenant guidance; accepts both chat and voice session channels |
| Administration and operations | services/api/tests/connectors.test.ts | stores per-provider message templates and rejects invalid mention settings; connects Intercom with regional OAuth and stores its access token encrypted |
| Administration and operations | services/api/tests/integrations-management.test.ts | lists safe metadata, validates without writes, rotates keys, and deactivates; isolates tenant listing and requires integration-management permission |
| Administration and operations | services/api/tests/llm-connections.test.ts | round-trips a secret and rejects a different key; masks all but the last four characters |
| Administration and operations | services/api/tests/llm-usage.test.ts | resolves a provider:model price and computes cost; ignores malformed price tables |
| Administration and operations | services/api/tests/llm-usage-limits.test.ts | reports no limits as unexceeded; warns at thresholds and flags over-limit usage |
| Administration and operations | services/api/tests/platform-auth.test.ts | throttles repeated failed operator sign-ins; issues an operator token for valid credentials |
| Administration and operations | services/api/tests/platform-audit.test.ts | filters by tenant and action and returns newest first |
| Administration and operations | services/api/tests/transcriber-policy.test.ts | inherits global defaults when the tenant has no preference; preserves tenant-owner preferences while the global field is in default mode |
| Administration and operations | services/api/tests/outbox-publisher.test.ts | leases each durable job to only one horizontally scaled publisher; releases failed jobs with exponential backoff for retry |
| Administration and operations | services/api/tests/retention.test.ts | purges expired transcript artifacts and objects; expires legacy source media with the transcript window by default |
| Connected journeys | services/api/tests/journey-e2e.test.ts | ingests voice, chat and email, completes analysis and calculates one correlated journey |
| Connected journeys | services/api/tests/audit-knowledge-flow.test.ts | requires explicit no-match confirmation and submits an all-unscored audit without a numeric result; accepts a valid pinned article section citation when the reviewer selects a possible match |
Additional inspected evidence includes packages/contracts/tests/contracts.test.ts; services/api/tests/integration/mongo.integration.ts; apps/web/src/app.test.tsx; apps/web/src/components/app-shell.test.tsx; services/api/src/manual-calibrations/alignment.test.ts; services/ai-runtime/tests/test_main.py; services/ai-runtime/tests/test_dispatcher.py; services/ai-runtime/tests/test_worker.py.
Checks performed for this change
Documentation validation is recorded in validation.json. It checks local links/anchors and source paths/line bounds against the pinned snapshot, all workflow links/IDs and required procedure fields, route-to-workflow coverage, Mermaid syntax using the Mermaid parser, and documentation-only change scope. The application was not started and existing application suites were not run for this documentation-only change. No provider request, credential validation, message delivery, production rollout or historical reanalysis was performed.
Operational acceptance scenarios still to run
- Sign in as each real local role; check permitted navigation, forbidden deep links and cross-tenant/ownership rejection, including the known coaching/live-assist discrepancies.
- In a configured non-production tenant, import an interaction, verify redacted evidence, allocate/reassign a review, autosave/reload, handle a stale revision, submit and verify the immutable result.
- Exercise correction, independent re-review, independent calibration/adjudication and manual group calibration with valid independent users and complete versus incomplete participants.
- Create/publish/revise knowledge and scoring definitions; confirm old pinned citations remain historical. Exercise attachment failure, stale draft and deletion dependency guards.
- Assign/complete training and coaching; check five-observation thresholds, duplicate/reset behaviour, closure notes and learner access limitations.
- With authorised provider test accounts, verify integration validation creates no domain work, credential rotation invalidates prior delivery and deactivation blocks new delivery.
- Verify Ask feature flags, private conversation ownership, explicit action confirmation, opt-in excerpt expiry and quality-review boundaries. Run paired evaluation release gates and a supported rollback.
- Observe worker retries, callback idempotency, provider/storage outages, cancellation and retention cleanup in the actual configured environment. Verify the separate transcriber against its pinned release.
Use the repository’s existing check and integration commands documented in its README/CI as applicable. Record actual outcomes with the tested commit and environment; do not infer success from these scenarios being listed.