Understanding Agentic AI: LLM Vulnerability Discovery

Definition of LLM Vulnerability Discovery

LLM vulnerability discovery is the systematic process of identifying, examining, and validating weaknesses, failure conditions, exploitable behaviors, or insufficient protections in a large language model (LLM) $M$ that may reduce the reliability, safety, robustness, or effectiveness of the model when it is used as the reasoning and decision-making foundation of an AI agent $A$. The process may examine how the LLM responds to different inputs $x$, operates under different execution conditions $c$, and produces model outputs $y=M(x\mid c)$ in order to identify circumstances under which its behavior becomes incorrect, inconsistent, unreliable, insufficiently protected, or otherwise undesirable. Such vulnerabilities may become particularly important in agentic applications because individual model weaknesses can affect planning, reasoning, coding, tool use, information processing, evaluation, and subsequent decisions across multiple execution steps. Systematically discovering and understanding these vulnerabilities can therefore provide evidence for improving the model, its surrounding safeguards, and the design of the agentic system, thereby increasing the efficiency and reliability of AI agents when performing both simple and long-horizon tasks $\tau$, including coding, professional work, and learning tasks.

Conditions for LLM Vulnerability Discovery to Hold

For LLM vulnerability discovery to hold, the process should systematically examine the LLM $M$ under sufficiently relevant and varied inputs $x$ and execution conditions $c$, observe the resulting model outputs $y$, and identify behaviors that provide meaningful evidence of a weakness, failure condition, exploitable behavior, or insufficient protection. A suspected vulnerability should be distinguishable from an isolated or incidental undesirable output by examining whether the behavior can be reproduced, consistently triggered under identifiable conditions, or otherwise supported by sufficient evidence to establish a meaningful weakness in the model or its use within an AI agent. The discovery process should also consider the practical consequences of the identified vulnerability for relevant tasks $\tau$, including whether it can impair reasoning, planning, coding, information processing, tool use, evaluation, decision-making, or other capabilities required for simple or long-horizon task execution. The process does not necessarily require every discovered vulnerability to have the same severity, cause, or remediation method; rather, it should establish a sufficiently clear relationship between particular conditions and undesirable model behavior so that the identified vulnerability can be analyzed, evaluated, and, where appropriate, addressed through model improvement, safeguards, agent design, or other mitigation measures.