Talk nerdy to me: Advanced LLM prompt injection techniques

Prompt injection attacks have evolved significantly alongside the increasingly widespread adoption of LLMs, yet many of the latest techniques are not used in security assessments. This talk explores why transformer-based models are fundamentally vulnerable to these attacks, and how emerging prompt injection techniques exploit these weaknesses in state-of-the-art (SOTA) models. We’ll examine new techniques drawn from recent research, discuss their effectiveness, and propose ideas for an updated prompt injection testing methodology. Attendees will walk away with practical insights to better assess and secure LLM applications.

As LLMs and their applications become more widely adopted, prompt injection attacks are also evolving, moving beyond simple "ignore all previous instructions" or "tell me a story like my grandma used to" payloads to more sophisticated techniques. However, many of these attacks remain underutilised in security assessments because they often remain undiscovered and confined to academic research, and are not incorporated into pen testing methodologies.

In this talk, we will briefly discuss the limitations of current LLM architectures, particularly the inability of transformer-based models to distinguish between system prompts and user inputs, which makes them fundamentally susceptible to injection-style attacks. Following this, we will explore emerging and advanced prompt injection techniques, focusing on how and why they work against SOTA and other widely used open-source models.

Given the non-deterministic nature of LLMs and the growing popularity of custom deployments, there is a need for a structured and updated approach to LLM security testing. We will discuss what a prompt injection testing methodology could look like, and why traditional web or API security practices fall short in this domain.

This methodology draws from recent research and is designed for security coverage of LLM applications. Firstly we'll discuss automated testing techniques, such as leveraging LLMs to test other LLMs, to identify potential weaknesses. These weaknesses can then be explored further through targeted manual testing using advanced prompt injection techniques discussed during the talk. We’ll also explore how LLMs are used in business applications, and how testing strategies can be implemented into existing testing methodologies in a way that does not treat LLMs as standalone components, but rather considers the whole system in its entirety, and results in more comprehensive coverage.