Continuous AI Security Regression in CI

• VP Engineering
• ML Platform or LLMOps Lead
• Application Security Lead
• QA Engineering Manager

LLM features change constantly: prompts are edited, providers upgrade models, tools are added and documents are re-indexed. Each change can quietly undo a fix from the last pentest. Without automated security tests in the pipeline, the first sign of a regression is often a customer report.

We build a security regression suite that runs in your CI pipeline whenever prompts, model versions, tool definitions or retrieval indexes change. The suite starts from your pentest and red-team findings and from probe sets for each relevant OWASP LLM category, written as versioned test cases with expected safe behaviour. Each case runs several times to account for non-determinism. Scoring combines deterministic checks, such as canary detection and forbidden tool calls, with an evaluator model calibrated against human-labelled examples. Results are compared with the last release, and new failures above an agreed severity block the merge. Borderline results go to a reviewer. Dashboards show trends by category and model version.

• Versioned test cases built from your past findings and OWASP LLM probe sets
• Triggers on prompt, model, tool, index or configuration changes
• Repeated sampling per case to handle non-determinism
• Deterministic checks plus an evaluator model calibrated on human labels
• Release gates on new failures above agreed severity
• Trend dashboards by category, model version and release

1. 01Build the baseline suitePast findings and probe sets are turned into versioned test cases with expected safe behaviour.
2. 02Trigger on changeCI runs the suite when prompts, models, tools, indexes or AI configuration files change.
3. 03Run with repeatsEach case runs several times against a staging deployment of the feature.
4. 04Score and compareDeterministic checks and the evaluator score each run, and results are compared with the last release.
5. 05Gate or reviewNew high-severity failures block the merge; borderline results go to a reviewer.
6. 06Track and extendResults feed dashboards, and new findings are added to the suite.

1. Test suite
   • Versioned test case repository
   • OWASP LLM probe sets
   • Cases from past pentest and red-team findings
2. CI integration
   • GitHub Actions or GitLab CI job
   • Change detection for prompts, models, tools and indexes
   • Staging deployment target
3. Execution and scoring
   • Parallel runner with repeated sampling
   • Deterministic checks for canaries and forbidden tool calls
   • Evaluator model with calibrated rubric
4. Gating and review
   • Severity thresholds and baseline comparison
   • Reviewer queue for borderline results
5. Reporting
   • Trend dashboard
   • Ticket creation in your issue tracker

Reviewers decide borderline results and confirm new failures before tickets are raised, and the security lead owns the severity thresholds. The evaluator model is recalibrated against human labels on a regular schedule.

• GitHub Actions, GitLab CI or Jenkins
• LLM gateway or provider APIs
• Issue tracker such as Jira or Linear
• Observability tools
• Slack or Microsoft Teams for alerts

• Suite runs against staging, never production users
• Evaluator agreement with human labels checked before gating is switched on
• Test secrets held in the CI secrets manager
• Cost and rate limits on each run
• Gate thresholds versioned and changed only by the security owner

• Continuous testing records help evidence ongoing monitoring under NIST AI RMF and ISO/IEC 42001.
• Test categories map to the OWASP Top 10 for LLM Applications.

• SaaS & AI Product Companies

  LLM Application Penetration Test (OWASP LLM Top 10)

  A scoped pentest of your LLM feature and the app around it: prompt injection, data disclosure, output handling, excessive agency and system-prompt leakage, with reproducible evidence.

• SaaS & AI Product Companies

  AI Agent Red-Teaming (Tool Misuse and Goal Hijack)

  Red-team your AI agents for goal hijacking, tool misuse, excessive agency, privilege abuse and memory poisoning, in a sandbox that records every tool call and decision.

• SaaS & AI Product Companies

  RAG Data-Leakage and Access-Control Testing

  Test whether your RAG pipeline leaks documents across users, roles or tenants through retrieval, caching, citations or logs, using seeded canary documents and role-based test personas.

• SaaS & AI Product Companies

  EU AI Act and ISO/IEC 42001 Evidence Pack

  Assemble testing records, risk assessments, oversight design and technical documentation for your AI features, mapped to EU AI Act and ISO/IEC 42001 requirements, ready for counsel and auditors.