Safety — PACT Standard assessment axis

What this axis protects.

Safety protects people, rights, services and organisations from unacceptable harm when AI behaves incorrectly, unpredictably, insecurely or outside its intended conditions. The assessment covers normal operation, reasonably foreseeable misuse, environmental change, component failure and deliberate attack.

A model can meet an average benchmark and still be unsafe. Harm may arise from rare errors, unequal error costs, weak integration, misleading interfaces, delayed human response, dependency on an unavailable supplier or an attacker manipulating data and prompts. A safety case must connect technical performance to the real consequence of failure.

A safe system has known limits, monitored operation, tested recovery and an organisation prepared to stop using it.

Map hazards before selecting metrics.

Output hazards. Incorrect, fabricated, incomplete, stale, harmful or overconfident outputs and the actions they can trigger.
Context hazards. Populations, languages, environments, edge cases and operating conditions not represented in development or validation.
Human-factor hazards. Automation bias, alert fatigue, poor controls, inaccessible interfaces, unclear uncertainty and pressure to use the system outside its competence.
Integration hazards. Incorrect permissions, tool use, data pipelines, cascading errors, brittle dependencies and failures at system boundaries.
Security hazards. Evasion, poisoning, extraction, privacy attacks, prompt injection, misuse, unauthorised actions and compromise of the AI supply chain.
Change hazards. Model, data, threshold, provider, policy or environment changes that invalidate earlier tests without a visible release event.

Where safety commonly fails.

One aggregate score. Performance is averaged across easy and hard cases, hiding the errors that carry the greatest harm.
Testing only the model. Evaluation excludes interface, integrations, operators, fallback, connected tools and downstream decisions.
Development data reused as proof. The same assumptions and data shape both the system and the evidence offered to validate it.
No misuse model. Teams test intended prompts or inputs but not plausible mistakes, adversarial behaviour or use outside stated boundaries.
Monitoring without thresholds. Logs and dashboards exist, but no result requires investigation, rollback, notification or suspension.
Incident response without AI detail. General IT procedures do not cover model rollback, harmful output containment, data poisoning, prompt or retrieval compromise and affected-person remedy.
Unsafe dependency. The service cannot function when the vendor, network or model is unavailable or changed without notice.

Assessment questions.

What harms are plausible for each use, affected group and system boundary, including misuse and malicious action?
Which harms are unacceptable regardless of probability, and who approved the remaining risk tolerance?
Do tests represent real populations, languages, edge cases, peak loads, degraded inputs and the conditions in which people will use the system?
Are metrics tied to the consequence of different errors, with uncertainty and subgroup results reported rather than one average?
How are security, privacy and misuse threats assessed across data, models, prompts, retrieval, tools, infrastructure and suppliers?
Which signals reveal drift, degradation, unexpected use or emerging harm, and what thresholds trigger action?
Can the organisation isolate, roll back, switch off or replace the system without causing a second harm?
How will affected people be notified, supported and provided remedy after a material failure?

Evidence to collect.

System definition: Intended purpose, users, affected people, architecture, data and model dependencies, interfaces, permissions, use boundaries and prohibited uses.
Hazard register: Hazard, cause, affected group, severity, likelihood, detectability, reversibility, control, owner and residual risk, including known and foreseeable misuse.
Evaluation plan: Independent and repeatable testing across performance, uncertainty, robustness, security, human factors, integration, accessibility and operational scenarios.
Test results: Disaggregated results, failed cases, confidence limits, benchmark relevance, red-team findings, control effectiveness and unresolved uncertainty.
Monitoring plan: Signals, baselines, review frequency, alert thresholds, owners, investigation route and explicit criteria for rollback, restriction or suspension.
Recovery evidence: Exercise results for containment, manual fallback, model or configuration rollback, supplier failure, notification, remedy and safe retirement.

Decision thresholds.

Proceed: Hazards are mapped to evidence and controls; tests represent real conditions; residual risk is explicitly accepted; monitoring and recovery have measurable triggers and tested owners.
Proceed with conditions: Uncertainty is bounded through limited population, purpose, duration, permissions or human confirmation, with a defined evidence plan and automatic expiry.
Stop or redesign: A plausible failure can cause unacceptable or irreversible harm; relevant performance cannot be measured; critical security or integration risks remain uncontrolled; or the system cannot be safely contained, rolled back or replaced.

Measure control effectiveness, not control presence. A filter, reviewer, warning or kill switch counts only after representative tests show which harm it prevents, where it fails and how quickly the organisation can act.

Lifecycle checks.

Before first use

Define intended conditions and unacceptable harms, complete independent evaluation, rehearse incident and fallback procedures, and restrict permissions to the minimum consequence the evidence supports.

During operation

Track performance, drift, unsafe outputs, override patterns, security signals, near misses and complaints. Investigate deviations against pre-agreed thresholds, not only after a public incident.

When conditions change

Require change control and revalidation for model, data, supplier, prompt, retrieval source, tool, permission, threshold, user group or environment changes. Monitor third-party changes that may occur without local deployment.

Sources and further reading.

Primary and institutional sources reviewed for this working document. Legal applicability must be assessed for the specific system and context.