Sit, Stay, Fetch: How to Train Your AI for ASPICE

If you conduct or contribute to ASPICE assessments, your default reaction to “AI-driven compliance” should be skepticism. Large Language Models are nondeterministic and prone to hallucinations - making the idea of a fully autonomous AI assessor a dangerous one. Nevertheless, ASPICE is surprisingly well suited for AI use cases due to its strict, rule-based structure. After experimenting with several approaches, we distilled our final framework for our ASPICE agent into the following guidelines.

Whitepaper

AI-Assisted ASPICE Compliance

How knowledge graphs and LLM agents automate ASPICE process assessment: two-pass consistency check, five specialized agent skills, and a cost-benefit analysis for a 5,000-requirement reference project.

Free · PDF

Download the whitepaper

Everything That Can Be Scripted Shall Be Scripted.

This is basically a variation of “use the right tool for the right job”. While it is tempting to try to just give the AI the right context and send it on its way with elaborate prompts, it is simply not the best strategy. Leverage the strict, rule-based nature of ASPICE process descriptions. Any metric or rule that can be defined deterministically, must be coded, ensuring compliance checks are based on objective, reproducible logic rather than probabilistic inference.

ASPICE-Example: The agent never assigns a rating itself: the N/P/L/F scale (see Automotive SPICE PAM v4.0 (Table 16 - Rating scale)) is a lookup in code - 0-15 % is Not achieved, 16-50 % Partially, 51-85 % Largely, 86-100 % Fully - and the skill forbids the model from doing the arithmetic itself, directing it to call the calculation tool instead. The model judges which artifacts count as evidence for a base practice; the code decides what percentage that makes and which letter it earns.

Persist Results and Progress.

Relying on AI memory creates an un-auditable black box. By persisting every step to a structured audit trail (e.g. JSON files), logging artifact attributes, model reasoning or scoring stages, the agent converts probabilistic LLM outputs into a transparent, fully inspectable workflow. Beyond auditability, keeping state outside the context window allows the pipeline to instantly resume from any checkpoint or cleanly re-evaluate specific steps without starting from scratch.

ASPICE-Example: The agent records every base practice with its artifacts, its gaps and each check’s sub-result together with the weight that sub-result carries, so any figure in the final report can be recomputed from what is on disk. It also notes, for each artifact, whether the revision safe storage system was entered by a human or inferred by a scan and writes its own step position to a separate state file, so a run resumes at the step it left rather than at the beginning.

Don’t Bite Off More Than You Can Chew.

Valid in real life and also with AI. If you have a significant number of artifacts and links - which is usually the case in the context of ASPICE - avoid running large-scale, complex evaluations on all of them simultaneously by breaking them down into manageable chunks (batches). Taken to the extreme: 1 LLM call per artifact or 1 LLM call per triple (subject, predicate, object). However, sending a separate prompt for every single artifact/triple wastes your token budget on redundant system instructions while crippling execution speed, so find the right balance.

ASPICE-Example: The agent processes artifacts in batches and ordered by ID so the batch boundaries fall in the same place on every run and computes the totals once, after the last batch, so no category ends up split across two of them. Documents are broken down the same way: the scanner works section by section rather than taking on whole files and a closing count check catches any batch that was never processed at all.

Don’t Ask Open-Ended Questions.

LLMs are trained to be “helpful”, so they tend to give detailed and comprehensive responses. Don’t let them. Not only with the usual “Give a short answer, please.”, but by forcing choices. Restrict outputs to binary answers wherever possible to reduce complexity and significantly lower the risk of hallucinations. Complex questions can often be split into smaller yes/no (or closed-set) sub-questions, which can then be logically combined to produce the desired answer.

ASPICE-Example: The agent is never asked whether two linked requirements are consistent. It answers closed checks in a fixed order and writes each answer down before going further, e.g.: “Are source and target in the same functional domain?” - yes or no; “Does the target cover all aspects of the source, fully, partly or not at all?”; “Do the two specify contradictory values?” - yes or no. A first-match table then maps that set of answers to exactly one finding type and a deterministic rating, so the verdict is combined by rule rather than composed by the model (see first guideline).

Establish What NOT To Do.

Give specific directives to the AI-model, formulated as prohibitions/negative imperatives: “Don’t do this! Never do that!”, etc.. Giving context, rules, decision records and the ultimate goal tells the model what to do and how to approach it step by step. But more often than not, this does not adequately rein in the model’s inherent “creativity” and tendency to roam free. Telling it what - absolutely, under no circumstances, never ever - to do helps.

ASPICE-Example: “Never compute ratings yourself!” (again, see first guideline), “Never fill a gap with a plausible value!” or, where a document is quoted as evidence, “Never simplify, paraphrase or summarize quotes!”

When In Doubt: Flag.

We tend to hold AI to higher standards than ourselves, but an AI-driven pre-assessment does not really require 100% precision. However, it demands near-perfect detection rate, catching every single actual compliance issue (zero false negatives), even if that means that it occasionally flags an artifact that is actually fine (false positive). And this conservative bias is the foundation for trustworthiness.

ASPICE-Example: The Type Check gives the agent an exit that is not a verdict: where an artifact does not clearly match one category, it must say so, reinforced by “Never force a guess!” (see previous guideline). And that answer is routed for human review, whereas a forced guess could produce a false pass.

Conclusion

Successfully integrating AI into ASPICE compliance is not about replacing the assessor, but about constructing a framework that keeps the AI within safe, deterministic bounds. By treating the rules proposed in this article as guardrails, we significantly improved the correctness and reproducibility of our results and managed to move away from the black box uncertainty of LLMs toward a system that is transparent, auditable and trustworthy.

By constraining an AI agent this heavily, one risks demoting it to an assistant. But the guardrails govern how it works, not whether it works on its own. It still conducts the assessment: drives the steps, gathers the evidence, asks when the data runs out and resumes wherever it left off. It just never signs the result. The assessor keeps the verdict - the agent does the work.

Requirements traceability in practice — How end-to-end traceability works in regulated projects, which tools have proven themselves, and when the RTM approach reaches its limits: Requirements Traceability at itemis →