Evaluation configuration

Manual home · Role guides · Coverage · Gaps

Manual conventions and recovery apply to every workflow. Evidence level: code inspected; related tests were inspected, not executed.

flowchart LR
  K[Published Knowledge revision] --> A[Published Atom v1]
  A --> M[Molecule draft]
  M --> P[Published Molecule version]
  P --> S[Scorecard draft]
  S --> V[Published Scorecard version]
  V --> R[Assignment for new work]
  A --> AD[Successor Atom draft]
  P --> MD[Successor Molecule draft]
  V --> SD[Successor Scorecard draft]

EVA-01 — Browse and create published Atoms

Status: Implemented. Verification: code inspected; no runtime completion claim.

Purpose: browse and create published Atoms.

Actors and access: qms:read to browse; molecules:manage for mutation handlers.

Prerequisites: A defined evaluation criterion with supported outcomes and any linked published knowledge revision.

Entry point: Atoms library.

1. Search/filter the library and inspect existing definitions and usage before adding duplicates.
2. Create an Atom with its criterion, evaluator and outcomes; verify any pinned knowledge source.
3. Save and inspect version one, which is published by Atom creation.

Result: A reusable published Atom is available for composition.

Exceptions and recovery: Atom creation differs from Molecule creation: it publishes v1. Source availability is validated; archived records must be restored before mutation. Review AI-created candidates using EVA-09 before saving.

Source evidence: services/api/src/atoms/routes.ts; services/api/src/atoms/lifecycle.ts; apps/web/src/features/atoms/atoms.tsx.

EVA-02 — Revise, publish, retire, archive and delete Atoms

Status: Implemented. Verification: code inspected; no runtime completion claim.

Purpose: revise, publish, retire, archive and delete Atoms.

Actors and access: molecules:manage; qms:read for usage/detail.

Prerequisites: An existing Atom; a published base for revision.

Entry point: Atom detail/version actions.

1. Create the next draft from a published version; edit the draft or discard it.
2. Publish the reviewed draft, or retire an eligible published version. Existing Molecules retain their pinned references.
3. Archive/restore the whole Atom; inspect usage before permanent deletion. Review every result when using bulk actions.

Result: Version history remains immutable while active library visibility can change.

Exceptions and recovery: Only one draft may exist. Published/retired definitions cannot be edited. Permanent deletion requires archive and no Molecule-version references; a bulk failure does not roll back successful items.

Source evidence: services/api/src/atoms/routes.ts; services/api/src/atoms/lifecycle.ts; apps/web/src/features/atoms/atoms.tsx.

EVA-03 — Create and edit a Molecule draft

Status: Implemented. Verification: code inspected; no runtime completion claim.

Purpose: create and edit a Molecule draft.

Actors and access: molecules:manage; qms:read for browsing.

Prerequisites: Published Atom references and an agreed grouping/evaluation purpose.

Entry point: Molecules → new/editor.

1. Browse the library and inspect Atom/Molecule usage.
2. Create a Molecule with its draft definition, evaluator/composition and metadata.
3. Edit permitted name/description/tags and draft content, then review the composed scoring criteria.

Result: A draft Molecule exists and can be reviewed before publication.

Exceptions and recovery: Validate pinned Atom references and supported outcomes/weights. Metadata changes are distinct from immutable version definition changes; archived Molecules must first be restored.

Source evidence: services/api/src/molecules/routes.ts; services/api/src/molecules/lifecycle.ts; apps/web/src/features/molecules/editor.tsx.

EVA-04 — Publish and maintain Molecule versions

Status: Implemented. Verification: code inspected; no runtime completion claim.

Purpose: publish and maintain Molecule versions.

Actors and access: molecules:manage.

Prerequisites: A valid draft, or published base for revision.

Entry point: Molecule detail/version pages.

1. Publish the reviewed draft; later create a successor draft from a published version and edit or discard it.
2. Retire an eligible published version when it should leave current use; inspect historical references.
3. Archive/restore the Molecule. Inspect Scorecard and alert-rule usage before permanent deletion or bulk lifecycle actions.

Result: Published scoring definitions remain pinned for historical evaluations.

Exceptions and recovery: Only drafts are editable/discardable and only one draft is allowed. Deletion requires archive and no Scorecard/alert-rule references; retirement does not rewrite historical scorecards.

Source evidence: services/api/src/molecules/routes.ts; services/api/src/molecules/lifecycle.ts; apps/web/src/features/molecules/molecules.tsx.

EVA-05 — Author and edit scorecard drafts

Status: Implemented. Verification: code inspected; no runtime completion claim.

Purpose: author and edit scorecard drafts.

Actors and access: scorecards:write; qms:read for browsing.

Prerequisites: Published Molecule versions and agreed scoring dimensions/weights.

Entry point: Scorecards → New / draft editor.

1. Create a scorecard and choose its scoring mode and published Molecule composition.
2. Edit the draft's sections, relative weighting and supported configuration.
3. Preview validation/scoring and save the draft before publication.

Result: A coherent version-one draft is stored.

Exceptions and recovery: Invalid weighting/outcomes or unavailable Molecule references prevent a valid publish. The builder is not a way to mutate an already published version; create a successor with EVA-06.

Source evidence: services/api/src/scorecards/routes.ts; packages/contracts/src/scorecards.ts; apps/web/src/features/scorecards/builder.tsx.

EVA-06 — Publish, revise and delete scorecards

Status: Implemented. Verification: code inspected; no runtime completion claim.

Purpose: publish, revise and delete scorecards.

Actors and access: scorecards:publish to publish; scorecards:write to revise/delete.

Prerequisites: A valid draft or an existing published scorecard.

Entry point: Scorecard detail and version actions.

1. Publish the selected draft after reviewing its immutable Molecule references.
2. For changes, create a successor draft from the latest published version, edit it, then publish explicitly.
3. Inspect feedback and dependencies before invoking permanent scorecard deletion.

Result: Published versions remain immutable; later assignments can select a new version.

Exceptions and recovery: Single-draft and published-reference checks apply. Deletion also considers audit usage and removes versions/assignments when permitted; treat it as permanent. Existing audits retain their original immutable scoring context.

Source evidence: services/api/src/scorecards/routes.ts; apps/web/src/features/scorecards/scorecards.tsx.

EVA-07 — Assign and resolve a published scorecard

Status: Implemented. Verification: code inspected; no runtime completion claim.

Purpose: assign and resolve a published scorecard.

Actors and access: scorecards:assign to create/change assignment; qms:read to list/resolve.

Prerequisites: An immutable published version and supported routing criteria.

Entry point: Scorecard assignment controls/API.

1. Create an assignment selecting version, priority and channel/queue/team/product/scenario/cohort criteria.
2. Resolve a representative interaction context and inspect the selected assignment.
3. Activate/deactivate the assignment as offered and recheck resolution.

Result: Deterministic priority/specificity rules select a published scorecard for new work.

Exceptions and recovery: Draft versions cannot be assigned. Existing claimed audits keep their pinned version; reassignment rules do not silently rescore history.

Source evidence: services/api/src/scorecards/routes.ts; packages/contracts/src/scorecards.ts; apps/web/src/features/scorecards/scorecards.tsx.

EVA-08 — Generate a scorecard proposal and create its draft

Status: Implemented. Verification: code inspected; no runtime completion claim.

Purpose: generate a scorecard proposal and create its draft.

Actors and access: scorecards:write to request/create draft; qms:read for proposal detail.

Prerequisites: Configured tenant LLM, usage allowance and suitable redacted interaction evidence.

Entry point: Scorecards → AI suggestions.

1. Choose evidence/filter criteria and request a proposal set.
2. Wait for generation, inspect proposed criteria and evidence, and edit the reviewable proposal.
3. Explicitly create a scorecard draft; inspect it in the normal builder and publish separately.

Result: Reviewed proposal content becomes a draft, without automatic production publication.

Exceptions and recovery: Unavailable providers, usage exhaustion or insufficient evidence require correction before retrying. Job success is not approval of content. Avoid publishing unsupported or weakly grounded criteria.

Source evidence: services/api/src/scorecard-suggestions/routes.ts; services/api/src/scorecard-suggestions/generator.ts; apps/web/src/features/scorecards/scorecards.tsx.

EVA-09 — Generate and save evidence-backed Atom suggestions

Status: Implemented. Verification: code inspected; no runtime completion claim.

Purpose: generate and save evidence-backed Atom suggestions.

Actors and access: molecules:manage.

Prerequisites: Configured LLM, usage allowance and selected interaction trends.

Entry point: Atoms → suggestions.

1. Choose trend filters and request an Atom suggestion set.
2. Monitor the job and inspect candidate evidence and definitions.
3. Edit the selected candidate and explicitly save it; inspect the resulting published Atom v1.

Result: Only a reviewed, explicitly saved candidate becomes an Atom.

Exceptions and recovery: The set must be ready and contain that candidate. Invalid/missing knowledge revisions fail validation. Saving an Atom follows its v1 publication semantics; it does not automatically insert it into existing Molecules.

Source evidence: services/api/src/atoms/routes.ts; services/api/src/atoms/suggestions.ts; apps/web/src/features/atoms/atoms.tsx.

EVA-10 — Generate a Molecule composition proposal

Status: Implemented. Verification: code inspected; no runtime completion claim.

Purpose: generate a Molecule composition proposal.

Actors and access: molecules:manage.

Prerequisites: Available Atoms/tags, a configured LLM and sufficient allowance.

Entry point: Molecule proposal controls/API.

1. Inspect tag options and request composition proposals.
2. Monitor the proposal job and review suggested Atom combinations and rationale.
3. Select/edit the proposal and explicitly create a Molecule draft; use EVA-04 to publish.

Result: A human-reviewed draft is created from a proposal.

Exceptions and recovery: A provider/job error does not create an approved definition. Validate source references and publish through the normal lifecycle; do not treat a proposal as a published Molecule.

Source evidence: services/api/src/molecules/proposals.ts; apps/web/src/features/molecules/molecules.tsx.

EVA-11 — Review, apply, retry or discard an AI enrichment proposal

Status: Feature-gated. Verification: code inspected; no runtime completion claim.

Purpose: review, apply, retry or discard an AI enrichment proposal.

Actors and access: Feature-specific permission and enabled enrichment kind; server checks the target operation.

Prerequisites: A supported target, current source revision, configured provider and usage allowance.

Entry point: Feature enrichment panel; ai-enrichments API.

1. Request an enrichment from the feature's panel and inspect job/proposal status.
2. Compare the proposed changes and target revision; select the supported changes.
3. Apply explicitly, retry a failed proposal, or discard it. Inspect the updated target and publish separately where its lifecycle requires.

Result: Accepted changes are applied through the target's validation boundary.

Exceptions and recovery: Stale targets, disabled kinds and unsupported transitions must be resolved before apply. Applying enrichment does not waive publishing gates or turn proposals into automatic decisions.

Source evidence: services/api/src/ai-enrichments/service.ts; apps/web/src/features/ai-enrichments/proposal-panel.tsx; packages/contracts/src/ai-enrichments.ts.