safety

Definition

The property that a system avoids harmful outputs and dangerous behaviour, pursued both during training and through runtime checks such as filters and monitors. It overlaps with two neighbours without being either: security assumes an adversary trying to break in or extract something, while alignment is about following intent — a perfectly aligned model given a harmful instruction is an alignment success and a safety failure. Papers are often loose about which they mean.

Explainers using this term

• Reward Hacking — Whatever You Measure Is Where It BreaksConcrete Problems in AI Safety