Administration and operations

Manual home · Role guides · Coverage · Gaps

Manual conventions and recovery apply to every workflow. Evidence level: code inspected; related tests were inspected, not executed.

flowchart TD
  A[Configure authorised source and policy] --> B[Validate or test explicitly]
  B --> C[Accept durable work]
  C --> D[Worker claims lease]
  D --> E{Processing succeeds?}
  E -->|Yes| F[Authenticated callback and domain update]
  E -->|Transient failure| G[Bounded retry and backoff]
  G --> D
  E -->|Exhausted or unsupported| H[Investigate failed or dead letter work]
  F --> I[Observe and apply retention]

OPS-01 — Onboard, validate, rotate and deactivate a generic integration

Status: Implemented. Verification: code inspected; no runtime completion claim.

Purpose: onboard, validate, rotate and deactivate a generic integration.

Actors and access: integrations:manage for management; delivery uses the issued channel-aware credential.

Prerequisites: Existing tenant team/agent defaults and a supported channel/payload.

Entry point: Settings → Integrations → generic voice/chat/email setup.

1. Create the integration with channel/default attribution; securely capture the newly issued credential shown once.
2. Run validation-only payload checks and inspect schema/redaction/attribution errors without ingesting records.
3. Deliver through INT-03 or the voice-result boundary; inspect source health. Rotate when needed or deactivate to stop future delivery.

Result: A managed integration authenticates supported delivery; validation creates no interaction/artifact/job.

Exceptions and recovery: Stored credentials cannot be retrieved later; rotation invalidates the prior credential. Deactivation does not erase previous interactions. Validate the correct channel and never place credentials or transcript/PII in health/error documentation.

Source evidence: services/api/src/integrations/management-routes.ts; services/api/src/transcription-results/routes.ts; apps/web/src/features/settings/integrations.tsx.

OPS-02 — Connect providers and map source attribution

Status: Implemented. Verification: code inspected; no runtime completion claim.

Purpose: connect providers and map source attribution.

Actors and access: Owner/platform override for setup, OAuth authorisation, tests and attribution; integrations:manage for protected connector reads.

Prerequisites: Provider-side access and deployment OAuth/encrypted-storage configuration where required.

Entry point: Settings → Integrations → connector setup.

1. Select a supported provider and enter its required setup fields. Zendesk and Intercom use their explicit OAuth flows.
2. Complete callback authorisation and test the connection; inspect connection status.
3. For Zendesk/Freshdesk, load the provider directory and map source teams/agents to active tenant teams/agent accounts.

Result: A tested connection and validated attribution mapping are stored.

Exceptions and recovery: OAuth callbacks/webhooks have their own state/signature authentication before tenant-route protection. An integration manager can open settings but owner-only connector actions may still fail. Do not infer that a connection test delivered or imported real data.

Source evidence: services/api/src/connectors/routes.ts; services/api/src/connectors/catalogue.ts; apps/web/src/features/settings/integrations.tsx.

OPS-03 — Import provider conversations/calls and sync CRM outcomes

Status: Implemented. Verification: code inspected; no runtime completion claim.

Purpose: import provider conversations/calls and sync CRM outcomes.

Actors and access: Owner/platform override plus connector access; inbound webhooks use provider authentication.

Prerequisites: Active connector with valid credentials, required mappings and supported source ID.

Entry point: Connector import/sync controls.

1. Select the provider-specific import: Zendesk ticket/call, Freshdesk ticket, Intercom conversation or Aircall call.
2. Submit the identifier/channel, inspect immediate or queued results, and follow resulting interaction/job links.
3. For Salesforce, run the supported outcome sync and inspect correlation-linked survey/outcome records.

Result: Provider records enter the common redacted interaction/outcome pipeline.

Exceptions and recovery: Provider access, expired authorisation, unsupported IDs and missing mappings require correction. Intercom queues work; acceptance is not completed import. Never assume a configured connector has continuous backfill or arbitrary-provider import support.

Source evidence: services/api/src/connectors/routes.ts; services/api/src/connectors/intercom-jobs.ts; services/api/src/connectors/salesforce.ts; apps/web/src/features/settings/integrations.tsx.

OPS-04 — Configure outbound work notifications and reporting feeds

Status: Implemented. Verification: code inspected; no runtime completion claim.

Purpose: configure outbound work notifications and reporting feeds.

Actors and access: Owner/platform override for configure/test-message/rotate; Power BI feed uses its dedicated reporting key.

Prerequisites: Supported provider configuration and an active tested connection.

Entry point: Connector setup/test-message; Power BI feed/rotation.

1. Configure supported Slack/Teams/Discord/Jira notification destinations and event options.
2. Explicitly send a test message only for supported providers (Slack/Teams/Discord), then inspect the outcome.
3. For Power BI, configure authorised feed access, page the results, and rotate the reporting credential when needed. Disconnect an obsolete connection explicitly.

Result: Supported outbound events/reporting can use the configured connection.

Exceptions and recovery: A test message is an external send; documentation verification does not perform it. Jira does not share the same test-message support. Rotating a reporting key requires updating authorised consumers; inspect event/job failures independently.

Source evidence: services/api/src/connectors/routes.ts; services/api/src/connectors/outbound.ts; apps/web/src/features/settings/integrations.tsx.

OPS-05 — Manage LLM connections, defaults and test jobs

Status: Implemented. Verification: code inspected; no runtime completion claim.

Purpose: manage LLM connections, defaults and test jobs.

Actors and access: members:manage for connection list; owner/platform override for configuration/testing actions and local web pages.

Prerequisites: Encrypted credential storage and an allowed provider/base URL/model.

Entry point: Settings → LLM connections; Labs → LLM testing.

1. Create/edit a connection using the secure setup form, select defaults and tenant LLM settings.
2. Run a supported connection test or chat/test job; monitor job status and inspect sanitised results.
3. Change defaults or delete an obsolete connection after checking dependent selections.

Result: Supported AI features resolve a configured target without exposing stored secrets.

Exceptions and recovery: Invalid/unsafe URLs, absent encryption configuration, unavailable models and usage exhaustion block work. Provider tests may incur usage; none are performed for this documentation audit. Deleting configuration does not rewrite job snapshots.

Source evidence: services/api/src/llm-connections/routes.ts; services/api/src/tenant-llm-settings/routes.ts; apps/web/src/features/settings/llm-connections.tsx; apps/web/src/features/labs/llm-testing.tsx.

OPS-06 — Inspect tenant AI usage, limits and model pricing

Status: Implemented. Verification: code inspected; no runtime completion claim.

Purpose: inspect tenant AI usage, limits and model pricing.

Actors and access: usage:read at tenant APIs; local settings pages are owner/platform gated.

Prerequisites: Recorded usage and configured pricing/limits.

Entry point: Settings → Usage / Pricing.

1. Choose the supported period and filters and inspect usage totals/breakdowns.
2. Review effective weekly/monthly usage allowance and available model pricing.
3. If requests are blocked, ask the authorised platform operator to review/reset/configure limits through OPS-08.

Result: Users with access can explain observed AI use and effective limits/pricing.

Exceptions and recovery: Tenant pricing screens are not billing/payment collection. Missing price data is distinct from free usage; limit resets do not erase historical usage. Do not assume a manager's usage:read grant guarantees web-page access.

Source evidence: services/api/src/llm-usage/routes.ts; services/api/src/llm-prices/routes.ts; apps/web/src/features/settings/usage.tsx; apps/web/src/features/settings/pricing.tsx.

OPS-07 — Manage tenants, local members and platform activity

Status: Implemented. Verification: code inspected; no runtime completion claim.

Purpose: manage tenants, local members and platform activity.

Actors and access: Separate authenticated platform operator session; not granted by tenant owner role.

Prerequisites: Provisioned operator and supported tenant authentication configuration.

Entry point: Platform login → tenants/detail/activity.

1. Sign in to the platform console and create or open a tenant; inspect provider and retention settings.
2. Apply supported tenant/member role/status/password changes, respecting owner safeguards and external-management boundaries.
3. Review platform audit events and sign out when finished.

Result: Control-plane changes are recorded in the platform audit trail.

Exceptions and recovery: Auth0 tenant membership is externally managed and not equivalent to local member administration. Tenant owner access does not create a platform operator session. Failed/unsupported configuration changes require operator correction, not tenant-header manipulation.

Source evidence: services/api/src/platform/routes.ts; services/api/src/platform/auth.ts; apps/web/src/features/platform/platform.tsx.

OPS-08 — Manage platform models, price versions and tenant usage budgets

Status: Implemented. Verification: code inspected; no runtime completion claim.

Purpose: manage platform models, price versions and tenant usage budgets.

Actors and access: Authenticated platform operator.

Prerequisites: Configured model/price/usage repositories and encrypted credential storage for system models.

Entry point: Platform → Pricing; tenant Usage controls.

1. Create/update supported platform model configuration and price versions, including required initial credentials where applicable.
2. Inspect tenant/platform usage totals and export permitted CSV.
3. Set tenant limits, reset a weekly/monthly allowance when explicitly intended, or remove obsolete model/price entries through supported actions.

Result: Platform-owned model availability, price resolution and usage enforcement settings are maintained.

Exceptions and recovery: Price and model deletion follow their endpoint validation. Resets affect allowance accounting rather than deleting historic usage. Missing configuration can return not configured; tenant read-only pricing is OPS-06.

Source evidence: services/api/src/platform/routes.ts; services/api/src/llm-prices/resolver.ts; services/api/src/llm-usage/limits.ts; apps/web/src/features/platform/pricing.tsx.

OPS-09 — Maintain global and tenant transcriber policy

Status: Implemented. Verification: code inspected; no runtime completion claim.

Purpose: maintain global and tenant transcriber policy.

Actors and access: Platform operator for global/tenant exceptions; workspace owner for permitted tenant defaults.

Prerequisites: Configured policy repository and known policy precedence/locks.

Entry point: Platform → Transcriber policy; tenant policy detail; Workspace transcription settings.

1. Inspect effective global policy and tenant overrides, including each field's source and lock state.
2. Update global policy or permitted tenant exceptions; clear all or a single override only when inheritance is intended.
3. Owners configure the unlocked workspace fields; submit a new job to use its resulting policy snapshot.

Result: New jobs receive resolved policy/settings snapshots with controlled precedence.

Exceptions and recovery: Global/tenant limits and locks can prevent owner edits. Existing jobs retain their snapshots. Clearing an override restores inheritance rather than applying an arbitrary default; use the managed-policy document for exact field semantics.

Source evidence: services/api/src/platform/routes.ts; services/api/src/transcriber-policy/resolver.ts; docs/transcriber-managed-policy.md; apps/web/src/features/platform/transcriber-policy.tsx.

OPS-10 — Inspect service health and recover queued processing

Status: Implemented. Verification: code inspected; no runtime completion claim.

Purpose: inspect service health and recover queued processing.

Actors and access: Health endpoints are public as composed; audits:manage for AI jobs; infrastructure/operator access for logs and queue recovery.

Prerequisites: Configured durable repositories, object storage and enabled required workers.

Entry point: Health endpoints; AI jobs; operational logs/queues.

1. Check Node liveness/readiness and metrics plus FastAPI readiness; correlate failures with request/job IDs.
2. Inspect job state, stage, attempts and leases. Restore unavailable dependencies and let bounded lease/backoff retries proceed.
3. Use supported domain retry actions for failed work; investigate exhausted/DLQ work and replay only with verified scope/idempotency.

Result: Operators can distinguish service availability, accepted work and successful domain completion.

Exceptions and recovery: Node LLM outbox and internal worker use leases/backoff; FastAPI dispatch can be local or FIFO SQS. Unsupported stages are dead-lettered and exhausted failures need investigation. Health alone does not prove every workflow works; do not hand-edit repository state to manufacture success.

Source evidence: services/api/src/ai-jobs/outbox-publisher.ts; services/api/src/jobs/internal-worker.ts; services/api/src/jobs/routes.ts; services/ai-runtime/src/sentinel_ai/main.py; services/ai-runtime/src/sentinel_ai/dispatcher.py; services/ai-runtime/src/sentinel_ai/worker.py.

OPS-11 — Apply retention and inspect cleanup failures

Status: Implemented. Verification: code inspected; no runtime completion claim.

Purpose: apply retention and inspect cleanup failures.

Actors and access: Owner/platform operator for policy configuration; service/operator for cleanup observation.

Prerequisites: A tenant retention policy and configured content deleter.

Entry point: Workspace/platform retention settings; server cleanup loop.

1. Set transcript/redacted-playback and unredacted source-media retention separately using the authorised settings.
2. Observe scheduled cleanup summaries/logs and check failed object deletions.
3. Restore storage access and allow the next bounded cleanup sweep to retry retained failed records.

Result: Expired transcript results/conversations and eligible stored artifacts are removed under policy.

Exceptions and recovery: Object-delete failures retain their artifact record for retry. Each artifact sweep is bounded at 500; one pass is not proof all expired content is gone. Permanent deletion is not reversible through the UI; avoid promising a backup restore procedure not verified here.

Source evidence: services/api/src/retention/cleanup.ts; services/api/src/server.ts; docs/transcriber-managed-policy.md.

OPS-12 — Request a retrieval-index rebuild

Status: Implemented. Verification: code inspected; no runtime completion claim.

Purpose: request a retrieval-index rebuild.

Actors and access: Owner/platform override.

Prerequisites: Configured job processing and tenant retrieval/index sources.

Entry point: Retrieval rebuild API.

1. Request the tenant rebuild through the owner-only endpoint.
2. Inspect the accepted job and monitor its completion/failure.
3. Validate accessible search/citation results against source records after completion.

Result: A tenant-scoped rebuild job refreshes supported retrieval data.

Exceptions and recovery: Acceptance is not completion and does not change source permissions. Missing worker/provider/index dependencies require remediation before retry. There is no documented dedicated rebuild UI in this baseline.

Source evidence: services/api/src/retrieval/routes.ts; services/api/src/retrieval/embedding-index.ts; services/api/src/server.ts.

OPS-13 — Inspect preview workspaces and supporting report records

Status: Incomplete. Verification: code inspected; no runtime completion claim.

Purpose: inspect preview workspaces and supporting report records.

Actors and access: reports:manage for report templates/schedules/runs; integrations:manage for health records; local intelligence UI gate also applies.

Prerequisites: Supported generic record schema and tenant data.

Entry point: Report templates; Integration operations; generic workspace API.

1. Open the preview/list and inspect stored templates, health or other available records.
2. Use authorised generic API create/update operations only for their supported records; report schedules can default timezone/next-run metadata.
3. Use actual report exports or connector health evidence for present operational needs.

Result: Stored records can be inspected/maintained; preview capabilities remain explicitly bounded.

Exceptions and recovery: No report execution/delivery worker was found for report-schedules/report-runs (GAP-03). A health record does not itself test a connection. Preview lists must not be presented as completed scheduling or delivery workflows.

Source evidence: services/api/src/workspace/routes.ts; apps/web/src/features/operations/operations.tsx; packages/contracts/src/operations.ts.