Cybersecurity and local AI: attacking yourself to defend

A personal lab can test itself the way an attacker would, with its own tools and models running on its own machines, then turn what it learns into measured defences. Here is the toolkit, the method and what model vendors publish.

By QDNA · Published on 30 October 2026 · 13 min read

Why attack yourself instead of waiting for the attack?

A defence written without knowing the attacks is like a lock fitted without knowing how doors get picked. Penetration testing therefore replays an adversary's moves against your own systems, within an authorised scope, to find weaknesses before someone else exploits them.

AI agents change the scale of this exercise, because a well-equipped agent chains reconnaissance, attempts and reporting without tiring, which helps attackers and defenders alike, so it becomes reasonable to hand part of the audit work to your own agents, as long as you keep control of the scope and trace everything.

Running the models locally follows the same logic of control, since logs, findings and attack traces describe a system's weaknesses, and they are exactly the data you do not want to hand to a provider, so the lab favours models hosted on its own machines and masks sensitive data when a task goes to an external model.

Which tools to test your own defences?

The lab's offensive core is called save-rs, a Docker-based penetration testing environment manager built in Rust and designed to work offline. Operations go through the Docker API rather than a shell, and all of its build targets forbid unsafe blocks outside tests.

The security of the tool itself matters as much as what it tests: sensitive data is encrypted at rest with AES-256-GCM and Argon2id, configuration files get owner-only permissions right after being written, and a SHA-256 hash-chained audit log keeps a verifiable record of every action taken during an engagement.

save-rs exposes its capabilities through MCP, the standard that lets an AI agent call tools. The 30 October 2026 count gives 1,446 tools exposed by the running server, wired into the lab's tool hub so that any local agent can call them under the same guardrails.

How do MITRE ATT&CK and ATLAS structure the toolkit?

MITRE ATT&CK is the public knowledge base of tactics and techniques observed in real attacks, each with its own identifier, while MITRE ATLAS is its counterpart for AI systems, with model-specific techniques such as data poisoning or prompt injection.

Mapping each tool to these frameworks gives a readable map of what the lab can test: in the tool descriptions, 172 tools cite at least one ATT&CK technique, for 177 distinct techniques, and 113 cite ATLAS, for 110 distinct identifiers, and since a single tool can cite several techniques, the counts differ.

A dedicated module gathers 26 red teaming tools for language models, designed as a defensive test bench. It builds on well-known open projects such as NVIDIA's garak, Microsoft's PyRIT and promptfoo, and files every test under an ATT&CK or ATLAS technique so that a defence team can measure its posture.

How does the toolkit learn from every attempt?

Every technique learned is recorded in a reusable card that the agent rereads before acting, which turns it into a building block of knowledge rather than a vague memory, and the lab holds 67 card families ranging from code audits to log monitoring and agent protocols.

These cards feed on public challenges. Hackropole, the training archive of ANSSI, the French national cybersecurity agency, which republishes the France Cybersecurity Challenge tasks, serves as a playground: 36 challenges are solved there with a matching flag, and each one produces a write-up, a lesson and, when the technique is new, a card.

To measure progress on harder tasks, the lab relies on Cybench, an academic benchmark of 40 professional capture-the-flag tasks that covers cryptography, web, reverse engineering and forensics, which forces the agent out of its comfort zone.

The last layer is a specialised local model, derived from a mid-sized open model and fine-tuned on the toolkit's traces with LoRA adapters. It runs on a single graphics card, without depending on any external service.

Where to practise without touching third parties?

Offensive tests never target third parties, since they run against a lab of deliberately vulnerable services, isolated in containers, and against challenges published for training. Every autonomous campaign runs in its own container, and a scope check verifies that every target was declared before acting.

The same toolkit can audit a real infrastructure, provided it stays read-only. The method is to split the domains between several models, local and remote, then cross-check their findings so that no single model can inflate a severity. The results of such an audit are not published: they are fixed.

How to defend: centralise, detect, repair?

The first rule of defence fits in one sentence: log everything in one place. A log store such as VictoriaLogs, reachable only from the machine that hosts it, receives the traces of the system, the probes and the agents, and keeps them long enough to trace an incident back several weeks later.

Detection relies on eBPF probes, small verified programs that the Linux kernel runs right where events happen. A tool such as Tetragon tracks program executions, privilege escalations and outbound connections, while a network filter can block, at kernel level, connections to addresses deemed dangerous.

This detection remains heuristic and produces false positives, often ordinary commercial services mistaken for leaks. Every corrected false positive becomes a regression test, so that it does not come back with the next update. Without this discipline, alerts end up ignored, which costs more than having no alert at all.

Automatic repair is deliberately restrained. A supervisor goes over the services at a regular interval and only allows reversible actions, such as starting a stopped unit, with a bounded number of repairs per pass. It never restarts a service that is already running and spaces out its attempts so as not to worsen an outage.

AI agents have their own guardrails too, because an agent that runs commands is an attack surface: dangerous deletions are filtered before execution, commands run under kernel confinement, and outbound requests are inspected for keys or secrets before they leave the machine.

The rest of the chain follows the same discipline. Secrets are served by a vault such as OpenBao rather than stored in plain text, the model gateway checks the fingerprints of its files before starting, and a filter masks personal data in requests sent to an external model.

Which limits to keep in mind?

Such a loop is never really closed, and saying so is part of the method. Consolidating alerts into a single stream, correlating sources and letting a local model read the detection logs remain hard tasks, which few setups, even professional ones, carry through to the end.

The most important link is also the most delicate: automatically turning a self-attack finding into a defence rule. Caution means a human validates this step, then a test that fails before the fix and passes after gates every deployment, with a rollback ready in case of failure.

What do vendors publish about their models' cyber capabilities?

In 2026, open-weight model vendors got into the habit of publishing their results on agentic cybersecurity benchmarks. CyberGym asks a model to reproduce real vulnerabilities found by fuzzing in open-source software, while ExploitGym and ExploitBench target the next step, exploitation, in environments built for evaluation.

These figures only compare with caution, because each vendor picks its own harness, reasoning effort and number of tasks. The following table therefore separates results published by the vendor itself from measurements by an independent evaluator, which come out lower in the comparable cases found here.

Independent evaluations confirm this caution. In April 2026, the US CAISI estimated DeepSeek V4 Pro's score at 32% on a subset of the 285 challenges of its CTF-Archive-Diamond benchmark. The UK AISI's simulated scenario has neither a defender nor alerts: it does not measure success on a real network.

Artificial Analysis' independent leaderboard on CyberGym-E2E shows the other side of the story. This benchmark asks a model to find a real memory-safety flaw in an open-source C or C++ project, trigger it, then fix it without breaking the project's tests, over 131 tasks of 90 minutes each. Kimi K3.1 Pro solves 94.7% of them.

Several closed models decline almost everything: Claude Opus 5.5, Claude Fable 5.1, GPT-6 Astra and GPT-6 Sol refuse 98 to 100% of the tasks on safety grounds, as does Qwen3.8 27B. With trusted access, the Daybreak Blue variant of GPT-6 Sol solves 93%, which shows that the refusal comes from the safeguards rather than from capability.

Closed-model vendors publish fewer scores and more risk levels. On 26 October 2026, OpenAI classified GPT-6.1 Terra as “Critical” in cybersecurity under its Preparedness Framework. On 13 October, Google observed elevated cyber capabilities in Gemini 4 Argon and deployed strengthened safeguards.

Google also states that Gemini 4 Argon solves all 12 end-to-end simulated cyberattack scenarios in its evaluation, up from five for the previous generation. On CWE-bench, the model card publishes 68% for Gemini 4 Argon and GPT-6 Astra, 67% for Claude Opus 5.5 and 58% for Claude Fable 5.1.

Anthropic presents Fable 5.1 as its most capable model in cybersecurity at release, on 1 September 2026. Yet it restricts the model on some dual-use tasks, in biology as in cybersecurity, and offers the same base with more permissive safeguards under the name Mythos 5.1.

For a defender, the consequence is concrete: open models close to the state of the art now run on local machines, which makes it possible to test yourself against a realistic level of capability without sending your data outside. The lab's model, fine-tuned locally, has no published score, hence the importance of in-house measurements on Hackropole and Cybench.

Sources: MITRE ATT&CK and MITRE ATLAS for the technique frameworks, ANSSI's Hackropole platform for the training challenges, Cybench (Zhang et al., arXiv 2408.08926) for the benchmark, and the VictoriaLogs documentation for log storage. For the models: the GLM-5.3 page at Z.ai, the DeepSeek V4.1 Flash model card, Averlon's evaluation, Fireworks' measurements, the AISI evaluation of Kimi K3, the NIST CAISI evaluation of DeepSeek V4 Pro, the MiniMax M3.1 Flash model card and the Mistral Large 4 announcement, all read on 30 October 2026. For closed models: OpenAI's GPT-6.1 Terra system card addendum, Google's post on Gemini 4 safety, the Google DeepMind model card for Gemini 4 Argon and Anthropic's Claude Fable 5.1 system card, read the same day. For the independent measurement: Artificial Analysis' CyberGym-E2E leaderboard and Moonshot AI's Kimi K3.1 Pro introduction, consulted on 30 October 2026.

Further reading

• Orchestrating multiple code agents in parallel: the supervisor agent
• Sensitive data and AI: what a zero-retention policy really guarantees
• Kill switch and local LLMs: what French parliamentary report No. 3054 changes