AlphaHack: Self-Improving Offensive Security Through Adversarial Self-Play

February 2026

Abstract

We present AlphaHack, a system for studying transfer learning in LLM-based offensive security agents. A Hacker agent attacks a curriculum of 80 capture-the-flag web application labs spanning 9 vulnerability classes at difficulties 1-10, building an episodic memory database through mandatory structured self-reflection after each engagement. We evaluate whether this memory enables measurable transfer learning by comparing the same agent’s performance with and without access to its accumulated experience.

Our initial experiment on a 20-lab held-out test set suggested memory provides only procedural speedup (51.5% fewer turns) without expanding capability (identical 64.7% solve rate). A follow-up retest on the full 80-lab training curriculum overturned this conclusion: memory-enabled solve rate was 67.3% versus 0% without memory, demonstrating that episodic memory is essential for capability, not merely efficiency. The discrepancy is attributable to test-set sampling bias.

Across both experiments, we observe a hard capability boundary: SQL­inject­ion (0% across 39 attempts regardless of memory) requires multi-step procedural reasoning that exceeds current agent capabilities. For vulnerability classes within the agent’s reach, 37 well-structured reflections are sufficient to achieve reliable expl­oitation.

1. Introduction

Large language models have demonstrated remarkable capability across software engineering tasks, but their potential for autonomous offensive security remains poorly understood. A core question is whether an LLM agent can learn from experience — not through weight updates, but through structured retrieval of its own past engagements.

AlphaHack tests this through adversarial self-play. A Builder agent generates vulnerable web applications of graduated difficulty. A Hacker agent attempts to expl­oit them using only HTTP requests. After each attempt, the Hacker fills out a structured reflection form capturing what it observed, hypothesized, tried, and learned. These reflections are embedded in a vector database and retrieved at the start of future engagements.

The central hypothesis: an agent that retrieves and applies structured offensive security knowledge from its own past experiences will demonstrate measurable transfer learning, solving labs with fewer steps and better strategy selection over time.

1.1 Contributions

1. A minimal, transparent architecture for LLM agent self-play (~4,000 lines of Python, no orchestration frameworks)
2. A structured episodic memory system based on mandatory self-reflection with a 20-field schema
3. Empirical measurement of transfer learning across 9 vulnerability classes and 10 difficulty levels
4. Evidence that episodic memory provides both procedural efficiency and capability expansion, contradicting our initial finding
5. Identification of a hard capability boundary (SQL­inject­ion) that memory cannot cross

2. System Architecture

2.1 Overview

┌─────────────────────────────────────────────────────────────┐
│                    ORCHESTRATOR                              │
│                                                             │
│  ┌───────────┐    ┌──────────┐    ┌────────────────────┐   │
│  │  Builder   │───>│Lab Files │───>│  Verifier Pipeline │   │
│  │  Agent     │    │(Flask)   │    │  (static + runtime)│   │
│  └───────────┘    └──────────┘    └────────┬───────────┘   │
│                                            │ PASS           │
│                                            v                │
│  ┌───────────┐    ┌──────────┐    ┌────────────────────┐   │
│  │  Hacker   │<──>│ Lab App  │<───│  Lab Manager       │   │
│  │  Agent     │    │(subprocess)   │  (lifecycle ctrl)  │   │
│  └─────┬─────┘    └──────────┘    └────────────────────┘   │
│        │                                                    │
│        v                                                    │
│  ┌─────────────┐    ┌──────────────────┐                   │
│  │  Reflection  │───>│  ChromaDB        │                   │
│  │  Form        │    │  Episodic Memory │                   │
│  └─────────────┘    └──────────────────┘                   │
│                                                             │
│  ┌──────────────────────────────────────┐                  │
│  │  Metrics Tracker (per-engagement)    │                  │
│  └──────────────────────────────────────┘                  │
└─────────────────────────────────────────────────────────────┘

Both agents are powered by Claude Sonnet 4 (claude-sonnet-4-20250514) via a simple while-loop over the Anthropic tool-use API. The agent loop is approximately 100 lines of Python with no dependencies on orchestration frameworks (LangChain, LlamaIndex, etc.). This transparency was a deliberate design choice: every tool call, every message, and every decision is directly observable.

2.2 Builder Agent

The Builder receives a vulnerability class, difficulty level (1-10), and flag placeholder. It generates a complete Flask web application with an embedded vulnerability. In practice, we used parameterized templates — one per vulnerability class — rather than free-form LLM generation, to ensure reproducibility and control lab quality. Templates produce:

• Realistic Flask applications with plausible UI and business logic
• Properly embedded vulnerabilities requiring active expl­oitation to extract the flag
• Difficulty-scaled defenses (input filtering, WAFs, red herrings, multi-step workflows)
• Flag placement appropriate to the vulnerability type (database table, file system, protected endpoint)

2.3 Hacker Agent

The Hacker receives a target URL and must extract the flag using four tools:

The Hacker’s system prompt enforces a five-phase methodology: Reconnaissance, Memory Retrieval, Hypothesis Formation, Expl­oitation, and Reflection. The agent is constrained to HTTP-only interaction — no shell­access, file system access, or network scanning.

2.4 Episodic Memory System

After each engagement, the Hacker fills out a structured reflection form (Pydantic model, 20+ fields) organized into five sections:

• Discovery: initial observations, tech stack identified, attack surface enumerated
• Hypotheses: ordered list of vulnerability hypotheses, each with a confidence score and result (confirmed/rejected/untested)
• Expl­oitation: successful technique, pay­load used, full expl­oitation chain
• Failure analysis: blocking factors, defenses encountered, closest approach
• Transfer learning: which retrieved memories helped, what was new, and an abstracted_lesson

The abstracted_lesson field is the critical design element. It forces the agent to distill lab-specific experience into a transferable insight. Examples from the experiment:

“When a Flask app renders user input in Jinja2 templates without escaping, test for SS​TI with { {7*7} } before trying more complex pay­loads.”

“SS​RF in Flask often requires http​://127.0.0.1 or file​:// protocols. Check for URL-fetch endpoints in /api routes.”

Reflections are embedded and stored in ChromaDB with cosine similarity search and metadata filtering on vulnerability class, difficulty, and outcome. At the start of each new engagement, the Hacker queries its memory with observations from initial reconnaissance and retrieves the top-5 most relevant past reflections.

2.5 Lab Verification Pipeline

Each lab passes through a six-step automated verification pipeline before the Hacker sees it:

1. Static checks: required files exist (app.py, Dockerfile, requirements.txt)
2. Dockerfile validation: CMD/ENTRYPOINT present, EXPOSE directive
3. Flag-in-HTML check: flag string not present in any static template
4. Runtime health check: app starts and responds to HTTP within 30 seconds
5. HTTP smoke tests: standard paths return non-500 responses
6. Flag leak detection: flag not present in any unauthenticated response across 13 common paths and 2 form submissions

This pipeline catches structural failures but does not verify expl­oitability — a limitation we discuss in Section 7.

2.6 Vulnerability Classes

Note: The IDOR template contained a schema bug (column name mismatch in init_db()) that caused all IDOR labs to fail verification. This eliminated the class from the experiment entirely.

2.7 Difficulty Scaling

Each template parameterizes difficulty across four dimensions:

3. Experimental Design

3.1 Experiment 1: Three-Phase Evaluation

The primary experiment runs three sequential phases:

Phases 2 and 3 use the exact same 20 lab specifications (same vulnerability classes, same difficulties, same random seed). The only variable is whether the memory database is accessible. This paired design provides a clean measurement of memory’s causal contribution.

Training curriculum (seed=42): Systematic cycle through vulnerability classes with linearly increasing difficulty. Lab i uses class VULN_CLASSES[i % len(classes)] at difficulty 1 + (i * 9) / 79.

Test curriculum (seed=999): Random selection of vulnerability class and difficulty to avoid training-order correlation.

3.2 Experiment 2: Controlled Retest

To address statistical power concerns with the 20-lab test set, we ran a follow-up retest using the full 80-lab training curriculum:

Phase B reuses the exact lab files generated in Phase A, eliminating any variation in lab implementation. The memory database contains the 37 entries built during Experiment 1’s training phase.

3.3 Metrics

3.4 Configuration

4. Results

4.1 Experiment 1: Three-Phase Results

Total wall time: ~3.3 hours across 105 completed engagements.

Initial interpretation: Solve rate is identical (64.7%) with and without memory. Memory provides a 51.5% turn reduction and 25.2% token savings, but does not expand capability.

Per-Lab Paired Comparison (Test Set)

Since Phases 2 and 3 attack the exact same labs, we compare head-to-head:

Memory won 9, No-memory won 2, Tied 6. Memory win rate: 82%.

4.2 Experiment 2: Retest Results

Phase A completed 55/80 labs before API credits were exhausted.

The retest overturns the initial finding. On the full training curriculum, the no-memory agent failed every single lab — including labs from vulnerability classes it solved 100% of the time in Experiment 1’s test set.

Per-Vulnerability-Class Breakdown

4.3 Per-Vulnerability-Class Training Results

The pattern is starkly binary: vulnerability classes are either near-fully solved or completely unsolved.

4.4 Turn Count Learning Curves (Training)

For solvable classes, tracking turns across increasing difficulty reveals learning:

Command Inject­ion: 11 -> 10 -> 11 -> 13 -> 44 -> 9 -> 12 -> 14 -> 10 (d1-d9)
Consistent ~10-turn performance except for a spike at d5. The agent internalized the pattern quickly.

Path Traversal: 22 -> 11 -> 10 -> 44 -> 12 -> 9 -> 9 -> 10 -> 9 (d1-d9)
Started slow (22 turns) but converged to ~9 turns from d6 onward. Classic learning curve.

SS​RF: 38 -> 10 -> 11 -> 10 -> 9 -> 9 -> 28 -> 14 (d1-d8)
Very slow start (38 turns), then rapid improvement. Spike at d7 suggests a harder variant.

SS​TI: 19 -> 17 -> 10 -> 8 -> 8 -> 8 -> 47 -> 31 (d1-d9)
Improvement from 19 to 8 turns, then d7+ defenses significantly increased effort.

4.5 Solve Rate by Difficulty (Training)

Solve rate is remarkably flat (~50%) because the dominant factor is vulnerability class, not difficulty. The agent either can or cannot expl­oit a given class; difficulty scaling within solvable classes does not meaningfully affect success rate.

4.6 Solve Speed Distribution

Median solve: 11 turns. Fastest: 8 turns. Slowest: 47 turns. The distribution is bimodal — the agent either converges quickly (8-15 turns) or thrashes until near the turn limit.

4.7 Token Economics

Estimated at $3/M input tokens, ~70/30 input/output split.

Unsolved labs are disproportionately expensive: they always consume the maximum 50 turns, burning ~500K tokens each. Failed engagements consumed approximately 74% of the total token budget.

5. Analysis

5.1 Reconciling the Two Experiments

The two experiments produced apparently contradictory results:

The resolution lies in test-set composition:

1. Experiment 1’s test set (20 labs, random seed=999) oversampled low-difficulty variants of solvable classes. With only 1-3 labs per class, the no-memory agent could stumble onto solutions through exploratory search within the 50-turn limit.

2. Experiment 2’s training set (80 labs, systematic seed=42) includes the full difficulty range for every class. Higher-difficulty variants require specific procedural knowledge (defense bypass strategies, encoding variations, multi-step expl­oitation chains) that the agent cannot reconstruct from scratch within the turn budget.

The corrected finding: On a representative curriculum with full difficulty coverage, episodic memory is essential for capability. The initial “speed only” conclusion was an artifact of a small, favorably-biased test set.

5.2 Why Memory is Essential

The memory database stores procedural knowledge that the base model cannot reliably reconstruct:

• Which endpoints to target for a given vulnerability class
• Which pay­load encodings bypass common defenses
• What the expl­oitation chain looks like end-to-end
• Which approaches are dead ends

Without these memories, the agent exhausts its 50-turn budget exploring dead ends. The turn-limit acts as a hard cutoff: given unlimited turns, the no-memory agent might eventually solve some labs, but the cost would be prohibitive (estimated 100+ turns per solve).

5.3 Memory Quality

37 well-structured reflections — roughly 3-4 per solvable vulnerability class — were sufficient to achieve 67.3% solve rate. This demonstrates:

• Quality over quantity: The abstracted_lesson field produces high-value retrieval targets
• Semantic retrieval works: ChromaDB’s cosine similarity successfully matches across similar vulnerability patterns
• Sparse coverage suffices: Complete coverage of every variant is not necessary; a few successful exemplars per class provide adequate guidance

5.4 The SQL­Inject­ion Boundary

The most significant negative result: zero SQL­inject­ion solves across 39 attempts (27 training + 12 test), regardless of memory, difficulty, or approach.

The failure mode is consistent across all attempts:

1. The agent correctly identifies SQL­inject­ion via error-based testing (' O​R 1=1 ​--)
2. It confirms SQ​Lite as the database engine
3. It attempts UNION SELECT but cannot reliably enumerate the column count
4. It enters a loop (UNION SELECT 1, UNION SELECT 1,2, …) without making progress
5. Even with correct column count, it struggles to extract data from the correct column position

This represents a multi-step procedural reasoning challenge: determine column count via ORDER BY, identify displayed columns, extract sqlite_master table names, query the target table. Each step depends on precise interpretation of the previous step’s output, and the LLM’s tendency to vary its approach between turns (rather than methodically iterating) makes this particularly difficult.

Memory cannot help because there are no successful reflections to retrieve. The failure is at the base capability level, not the retrieval level.

Implication: Certain expl­oitation techniques require algorithmic precision that may be better served by specialized sub-routines than by free-form LLM reasoning.

5.5 The Auth Bypass Paradox

Auth bypass exhibits an inverted difficulty curve: 0% solve rate at difficulties 1-6, then 100% at difficulties 7-9.

At low difficulties, the template generates simple cookie-based authentication. The vulnerability is subtle — the app looks “normal” and the agent fails to identify it as expl­oitable. At high difficulties, defenses are more complex but also more visible: JWT tokens, debug headers, and admin panels that signal where to attack.

The agent is better at attacking visible complexity than at finding subtle simplicity.

5.6 Training Progression

A modest upward trend in solve rate (40% to 60%) and general downward trend in turns. The progression is limited because ~38% of training labs are SQ​Li variants that never get solved, creating a floor effect.

6. Key Findings

Finding 1: Episodic memory is essential for capability on representative workloads

On the full 80-lab curriculum, memory-enabled solve rate is 67.3% versus 0% without memory. This is not a speed optimization — it is a capability requirement.

Finding 2: Small test sets can produce misleading conclusions

The initial 20-lab test set suggested memory only affects speed (identical 64.7% solve rate). This was an artifact of favorable sampling. Test-set size and composition must be carefully controlled in agent evaluations.

Finding 3: Vulnerability class dominates difficulty as a predictor of success

Solve rate is flat across difficulty levels 1-9 (~50%). The agent either can or cannot expl­oit a given vulnerability class. Within solvable classes, difficulty primarily affects turn count, not outcome.

Finding 4: Transfer learning is real but class-bounded

For vulnerability classes where the agent has successful experience, memory provides: 51.5% turn reduction (Experiment 1), reliable first-attempt success (Experiment 2), and consistent strategy selection. For classes the agent never solved, memory contains only failure data, which provides negative signal but not capability.

Finding 5: Multi-step procedural reasoning has hard limits

SQL­inject­ion — requiring precise sequential column enumeration, table discovery, and data extraction — produced 0% across 39 attempts. This capability boundary is not addressable through memory, training volume, or difficulty adjustment.

Finding 6: Structured self-reflection produces effective retrieval targets

The mandatory reflection form, particularly the abstracted_lesson field, generates embeddings that enable effective semantic retrieval. 37 entries (~3-4 per solvable class) are sufficient for reliable expl­oitation.

7. Limitations

7.1 Template Homogeneity

Each vulnerability class has a single parameterized template. The Hacker may partially memorize template structure rather than learn generalizable expl­oitation. Human-authored or multi-model-generated labs would provide a stronger test of generalization.

7.2 IDOR Template Bug

A column name mismatch in the IDOR template’s init_db() function caused all IDOR labs to fail at startup, eliminating the entire class from both experiments. The verifier caught the deployment failure but not the root cause (schema mismatch).

7.3 Verification Gap

The verification pipeline confirms labs boot and don’t leak the flag, but does not verify that vulnerabilities are actually expl­oitable. Labs can pass verification yet be unsolvable due to implementation errors.

7.4 HTTP-Only Constraint

The Hacker can only interact via HTTP. This makes timing-based techniques (blind SQ​Li via response time) impractical and eliminates out-of-band attack channels (DNS exfil­tration, reverse­shells). A broader toolset might change the SQ​Li results.

7.5 Same-Model Self-Play

Both the templates and the Hacker use Claude Sonnet 4. The Hacker is attacking applications that share its “mental model” of what vulnerable code looks like. Cross-model evaluation would be a stronger test.

7.6 Incomplete Retest

Experiment 2, Phase A completed only 55/80 labs due to API credit exhaustion. The 67.3% solve rate is based on this subset. However, the per-class pattern (100% on solvable classes, 0% on SQ​Li) is clear even with partial data.

7.7 Sequential Test Conditions

Both test conditions were run sequentially, not interleaved. If API response quality varies over time, this could introduce systematic bias. Randomized interleaving would be more rigorous.

8. Future Work

1. Fix the IDOR template and re-run with all 11 vulnerability classes
2. Human-authored test labs sourced from CTF platforms to eliminate template-familiarity confounds
3. Specialized sub-routines for SQ​Li to test whether memory helps once base capability exists
4. Cross-model evaluation: test with GPT-4, Gemini, and open-source models to measure model-specific vs. architecture-specific effects
5. Scale to 500+ labs for statistical power at the per-class level
6. Ablation studies: memory with only positive outcomes, only structured fields (no abstracted_lesson), random retrieval, and cross-agent memory transfer
7. Adversarial Builder mode: share the Hacker’s weaknesses with the Builder to create targeted challenges
8. Comprehensive verification: automated expl­oit validation to confirm labs are solvable before hacker engagement

9. Broader Implications

AlphaHack demonstrates that retrieval-augmented experience is a viable path toward self-improving AI agents, with a critical caveat: the base model must already possess the capability being augmented. Memory amplifies existing competence; it does not create new competence. The SQ​Li results show a hard capability boundary that structured self-reflection cannot cross.

For AI safety, the implications are nuanced. An agent with episodic memory becomes more capable and efficient at tasks within its reach, but does not spontaneously acquire fundamentally new dangerous capabilities through self-play alone. The boundary between “memory-amplifiable” and “memory-resistant” capabilities deserves further study.

For offensive security applications, the results suggest that LLM agents can be effective at vulnerability classes that require pattern recognition and creative pay­load construction (SS​TI, SS​RF, command inject­ion), but struggle with expl­oitation techniques that require precise multi-step algorithmic reasoning (SQL­inject­ion data extraction). This aligns with broader observations about LLM strengths and weaknesses in procedural tasks.