05 / 08

Safety, Security, Robustness and Reliability

AI should be tested for accuracy, dependable performance, resilience and security, with safeguards proportionate to foreseeable harm and the consequences of failure.

Safety, Security, Robustness and Reliability

5.1 Safety and prevention of unacceptable risk

Artificial intelligence shall be designed, deployed and operated so that foreseeable hazards are identified and controlled before they result in unacceptable harm. Harm includes physical, psychological, economic, legal, reputational and social injury, as well as harm to groups, communities and institutions. Safety requirements shall be defined in relation to the actual environment, affected persons, connected systems and reasonably foreseeable misuse, rather than only normal or intended operation.

5.2 Accuracy and precision

AI systems shall be evaluated against performance criteria appropriate to their intended use and the consequences of error. For generative AI, evaluation shall address, where relevant, factual accuracy, grounding in reliable sources, completeness, relevance, consistency and source fidelity, rather than relying on a single aggregate metric.

5.3 Reliability and consistency

An AI system shall provide sufficiently stable and dependable performance over time. Comparable inputs and circumstances shall not produce materially inconsistent outcomes without a justified reason, particularly where the output contributes to legal, professional or administrative decisions. Where variability is inherent in the system, the organization shall control and communicate it and shall establish procedures that prevent arbitrary treatment.

5.4 Robustness and resilience

Artificial intelligence shall tolerate foreseeable data errors, unusual inputs, environmental changes and partial failures without producing unacceptable consequences. The system shall recognize conditions that differ materially from those for which it was validated and shall not continue to produce confident output when its assumptions no longer hold. Organizations should maintain the ability to continue essential functions when an AI system or external service is unavailable, compromised, or materially changed.

5.5 Cybersecurity and access security

AI systems, models, datasets, interfaces, credentials and supply-chain components shall be protected against unauthorized access, disclosure, extraction, alteration and disruption. Security controls shall correspond to the sensitivity of the information, the importance of the function and the consequences of compromise. Security should cover both conventional cyber threats and risks specific to AI-enabled systems.