Tiered Governance Model for AI Usage in the Safety Lifecycle

ISO/IEC 42001 provides the organizational scaffolding for classifying AI usage: AI policy requirements (A.2.2), role allocation (A.3.2), resource documentation including data and tooling (A.4.2–A.4.4), and objectives for responsible AI use (A.9.3) together define what the organization must document and govern about its AI usage.

ISO/IEC 5338's lifecycle stages model (Figure 2: Inception → Design → Verification → Deployment → Operation → Continuous Validation → Re-evaluation → Retirement) helps distinguish where in the process AI is being used — a key factor in classification.

ISO/IEC TR 5469 introduces usage levels of AI technology (Table 1 in Clause 6) that classify AI by its final impact on the system. This maps closely to the layer concept: different usage levels trigger different safety assurance requirements. The classification considers whether AI technology forms part of a safety function, supports safety analysis, or is used in development tooling.

Gap to fill: None of the standards provide a ready-made "decision tree" for classifying real-world AI usage into governance tiers. Our model needs to synthesize the abstract standard requirements into concrete, actionable classification criteria that a project manager can apply.

The standards provide review mechanisms but don't specifically address layer migration. ISO 26262 Part 2's confirmation measures and safety planning give us the gate structure where reclassification checks can be inserted. During phase gate reviews and milestone checks, an additional AI-specific checklist should be followed, for example: "Has anyone used AI to work on, develop, or create any of the work products under review?", "Was the AI tool internally developed?", "Was it used/shared by multiple people in the team or by other teams?"

ISO 26262 Part 2 Annex B's safety culture guidance explicitly calls for continuous improvement and lessons learned as integral to all processes. This directly supports making AI usage a standing topic in lessons-learned reviews — organizations should systematically capture how AI was used in each project phase and what issues emerged.

ISO/IEC 42001's review mechanisms (A.2.4 AI policy review at planned intervals, management review per Clause 9.3) provide the organizational review cadence. ISO/IEC 5338's continuous validation process (6.4.14) provides a lifecycle-aware review mechanism.

Original contribution needed: Specific trigger definitions (what event causes reclassification), the AI-specific checklist items for gate reviews, and the process for promoting/demoting tools between layers. This is where our model adds the most original value.

Governance layer: ISO/IEC 42001 provides the foundation — AI policy (A.2.2) must guide all AI usage, roles and responsibilities must be allocated (A.3.2), and a process for reporting concerns must exist (A.3.3). The policy must explicitly address what AI usage is permitted, what must be declared, and what is prohibited.

Process layer: ISO/IEC 5338 emphasizes audit trails for safety-related AI systems, including data provenance, model traceability, and decision documentation. This must be extended to cover not just AI systems being developed but AI tools being used in the development process. Work product traceability, toolchain identification including AI detection, and clear methodology descriptions in processes are needed.

IT control layer: This is less covered by the standards in scope and falls more into cybersecurity territory (ISO 21434, currently out of scope but designed to extend naturally). Practical measures include: approved AI tool lists, network-level monitoring of AI service usage, endpoint controls, and data loss prevention systems.

Safety culture connection: ISO 26262 Part 2 Annex B's safety culture table is remarkably relevant. It contrasts organizations where processes are ad-hoc and implicit versus those with defined, traceable, controlled processes. It flags "groupthink" and suppression of dissent as poor safety culture indicators. Applied to shadow AI: an organization where employees fear disclosing AI usage (because it might be seen as "cheating" or "not doing their job") has a safety culture problem. Disclosure must be encouraged, not penalized.

Interaction with functional safety: Shadow AI is where the biggest risk hides. The safety evidence chain (ISO 26262's core assurance mechanism) assumes that every work product is traceable, reviewable, and produced by qualified people using qualified methods. Undeclared AI breaks this silently. Error propagation is the key danger: an AI-introduced flaw in early-phase work products (e.g., requirements) can survive all downstream V&V because tests verify against the flawed requirement, not against the actual need.

The same AI tool used in different contexts has vastly different risk profiles. An engineer using AI to format meeting notes (supporting activity) vs. using AI to derive software safety requirements (V-cycle core activity at ASIL D) cannot be governed the same way.

V-cycle phase progression: A critical insight is that AI usage becomes more dangerous as the project advances through the V-cycle. Using AI in early concept phases (hazard analysis, safety concept) is less critical because multiple subsequent verification layers exist to catch errors. Using AI in late phases (final validation before tape-out, safety case argumentation) is much more critical because fewer — or no — subsequent checks remain to catch AI-introduced errors.

ISO 26262 Part 2 Table 1 already defines which confirmation measures apply at which ASIL level, with which independence requirements. Part 8 Clause 11 considers the ASIL of safety requirements that could be violated if a tool malfunctions when classifying Tool Confidence Level. These existing mechanisms map well to the criticality axis.

ISO/IEC TR 5469 Clause 9 makes a critical observation: for AI-based systems, standard verification and validation is undermined because the function behavior progressively moves away from rigorously tested, deterministic behavior. This directly supports the argument for increasing scrutiny of AI usage in later V-cycle phases.

Needed: A concrete mapping — "at this V-cycle phase, with this ASIL, AI usage at Layer X requires these specific controls." This is a matrix that the standards support structurally but don't populate for the AI case.

ISO 26262 Part 8 Clause 11 provides the full traditional tool qualification framework. Tools are assessed for Tool Impact (TI: can a malfunction introduce or fail to detect errors?) and Tool error Detection (TD: can errors be detected/prevented?). These combine into a Tool Confidence Level (TCL1/2/3). TCL2 and TCL3 require qualification via one or more methods: increased confidence from use, evaluation of development process, validation, or development per a safety standard.

The AI problem: This framework assumes deterministic tools with stable, characterizable behavior. AI tools fundamentally break this: output varies with prompt wording, the model's internal state is opaque, there's no stable "function" to validate, and the tool may behave differently after a vendor update. "Increased confidence from use" is questionable when the same input can produce different outputs. "Validation" is challenging when you can't define expected outputs for all use cases.

The Part 10 alternative path: ISO 26262 Part 10 Clause 13 describes an alternative: instead of qualifying the tool, increase the probability of detecting erroneous tool outputs by adding process measures (review of tool output, additional test steps, checks by subsequent tools). This reduces TD to TD1, achieving TCL1 without direct tool qualification. This is highly relevant for AI tools — instead of trying to qualify the AI itself, qualify the human review process that checks its outputs.

What 5469 adds: ISO/IEC TR 5469 explicitly acknowledges that AI technology is not covered by mature functional safety standards and that continuous model improvement undermines V&V. It provides an extended catalogue of measures for AI-specific systematic failures (Annex A) and addresses transparency, explainability, and verification techniques for AI systems (Clause 9). This is the bridge between the traditional Part 8 framework and the AI reality.

Key question to resolve: What does "increased tool error detection" mean concretely when the tool output is a generated requirement, a generated test case, or a design suggestion? We need to define specific TD measures for each type of AI tool output in the safety lifecycle.

ISO/IEC 42001 A.10.2 and A.10.3 require that responsibilities are allocated between the organization, suppliers, and third parties throughout the AI system lifecycle, and that supplier products align with the organization's responsible AI approach. The implementation guidance in B.10.3 discusses different supplier types, varying risk levels, and the need for ongoing monitoring. Organizations should document how AI system components from suppliers are integrated.

ISO 26262 Part 8 §11.4.2 requires that predetermined tool qualification be re-verified before each project use. §11.4.6.2 requires documenting tool version, configuration, and environment — re-qualification is needed when any of these change. Part 10 Clause 13 explicitly warns about the high re-qualification effort for frequently changing tools.

Proposed two-path approach:

Path A — Controllable vendor: If the vendor or tool setup allows locking the model version and controlling settings, the problem is manageable. Qualification can be performed for a specific model version + configuration. Regression testing using a "golden dataset" (known inputs with verified correct outputs) should be executed before and after any model update to confirm performance stability. Re-qualification is triggered by model/version changes, at defined intervals, and at project milestones.

Path B — Uncontrollable vendor: If the vendor updates models daily with no version control, no notification, and no rollback capability, then tool qualification in the traditional sense becomes meaningless — you'd be qualifying a moving target. In this case, the only reliable approach is mandatory full human review/verification of every AI-generated output before it enters the safety lifecycle. This effectively treats the tool as an unqualified aid and relies entirely on process-level error detection (TD1 approach from Part 10).

ISO/IEC 5338 acknowledges that AI system lifecycle stages can be managed by separate organizations and addresses supply chain risks across organizational boundaries. This supports the need for clear contractual requirements about model versioning, update notification, and qualification support.

The core risk scenario: A software architect uses AI to help derive software safety requirements from the technical safety concept. The AI introduces a subtle misinterpretation — perhaps mapping a hazard to the wrong ASIL, or generating a requirement that sounds correct but misses an edge case. The architect reviews it, it looks reasonable, it goes forward. Downstream design, implementation, and testing are all based on this flawed requirement. Every subsequent review and test passes because they're testing against the wrong requirement. The defect survives to validation because the source was contaminated invisibly.

Why AI errors are different from human errors: ISO/IEC TR 5469 Clause 9 explains that data-driven models introduce fundamentally different failure modes: what's not in the training data can't be learned, what is in the data is learned but not always perfectly, and incorrect labels are a primary cause of learning errors. Applied to development tools: AI-generated errors tend to be plausible-sounding (they're trained on correct-looking text), internally consistent (the AI hallucinates coherently), and systematically biased (the same blind spots appear repeatedly) rather than random.

ISO 26262 Part 2 Annex C provides detailed confirmation review guidance for each type of work product (HARA, safety concept, integration strategy, safety case, etc.). ISO/IEC 42001 B.9.3 requires human oversight mechanisms including authority to override AI decisions and performance monitoring.

What needs to be added: AI-specific checks for each existing confirmation review point. For example, the HARA confirmation review (Annex C.3) should additionally ask: Was AI used in the situation analysis? Were AI-generated hazardous events cross-checked against domain expertise? Were ASIL classifications independently verified by a human who did not see the AI output first? These additions make the confirmation measures effective as a safety net against AI contamination.

The standards provide structural frameworks: ISO/IEC 42001 uses the Plan-Do-Check-Act management system cycle. ISO/IEC 5338 provides lifecycle process structure. ISO 26262 Part 2 provides safety plan structure and confirmation measure scheduling. None of them, however, provide the kind of quick-reference classification and decision tool envisioned for this model.

The operationalization needs to serve two audiences: management (who needs to understand risk exposure, resource implications, and governance obligations) and individual contributors (who need to know "what am I allowed to do, and what do I need to declare?").

Possible formats: A decision flowchart ("Is the output a formal work product? → Is it safety-relevant? → What ASIL? → Is the AI shared or personal? → Here are your obligations"), reference cards per layer, integration into existing safety plan templates, and checklists embedded in gate review procedures.

The goal is something a project manager can use on Monday morning when an engineer says "I've been using ChatGPT to help with the FMEA" — the model should immediately tell them what layer that falls into, what standards apply, and what corrective actions are needed.

All three primary standards allow risk-proportional application. ISO/IEC 42001 explicitly allows organizations to elect a risk-based approach to ensure appropriate control levels for particular AI use cases. ISO/IEC 5338 notes that organizations can tailor lifecycle processes. ISO 26262 has tailoring provisions throughout.

The strategy is to build the complete model first — viable for very large and complex cases (Tier 1 suppliers with thousands of engineers, ASIL D products, complex distributed development). Then, customization and simplification guidelines can be developed for smaller organizations, lower-ASIL products, and simpler development setups. This is where Eskube's consulting expertise in deploying and customizing processes comes into play.

Deferred to later phase: Customization guidelines are an application/deployment concern. The full model must be correct and complete first.

AI tools evolve extremely fast. A layer classification or tool qualification made today may be obsolete in months because the tool gained new capabilities, the underlying model changed, or new tools emerged. The governance model needs a built-in review cadence.

ISO/IEC 42001 A.2.4 requires AI policy review at planned intervals. ISO 26262 Part 8 §11.4.2 requires re-verification of tool qualification when the tool version or environment changes. ISO/IEC TR 5469 §8.7 discusses technology maturity as a risk factor that requires continuous monitoring, noting that even mature technologies need ongoing risk monitoring based on field data.

Minimum cadence: Layer classifications and tool qualifications should be reviewed at every project phase gate/milestone. For tools with controllable vendors, additional reviews should be triggered by model/version updates. For rapidly changing tools, a fixed-interval review (e.g., quarterly) should supplement milestone-based reviews.

This topic is closely coupled with Topics 2 and 6 and should be developed together with them.