Non-Deterministic Nature of Prompt Injection
As we explained in a previous blogpost, exploiting a prompt injection attack is conceptually easy to understand: There are previous instructions in the prompt, and we include additional instructions within the user input, which is merged together with the legitimate instructions in a way that the underlying model cannot distinguish between them. Just like what […]
Exploring Overfitting Risks in Large Language Models
In the following blog post, we explore how overfitting can affect Large Language Models (LLMs) in particular, since this technology is used in the most promising AI technologies we see today (chatGPT, LLaMa, Bard, etc). Furthermore, by exploring the likelihood of inferring data from the dataset, we will determine how much we can trust these […]
Using Semgrep with Jupyter Notebook files
If you frequently deliver source code review assessments of products, including machine learning components, I’m sure you are used to reviewing Jupyter Notebook files (usually python). Although I spend most of my time reviewing the source code manually, I also use static analysis tools such as semgrep, using both public and private rules. This tool […]
Project Bishop: Clustering Web Pages
Written by Jose Selvi and Thomas Atkinson If you are a Machine Learning (ML) enthusiast like us, you may recall our blogpost series from 2019 regarding Project Ava, which documented our experiments in using ML techniques to automate web application security testing tasks. In February 2020 we set out to build on Project Ava with […]
Exploring Prompt Injection Attacks
Have you ever heard about Prompt Injection Attacks[1]? Prompt Injection is a new vulnerability that is affecting some AI/ML models and, in particular, certain types of language models using prompt-based learning. This vulnerability was initially reported to OpenAI by Jon Cefalu (May 2022)[2] but it was kept in a responsible disclosure status until it was […]
How-to: Importing WStalker CSV (and more) into Burp Suite via Import to Sitemap Extension
In this post we show how to import WStalker output into Burp Suite and the Logger++ extension to build a sitemap from a recorded session for use in Intruder and Repeater.
Tool: WStalker – an easy proxy to support Web API assessments
Have you ever faced a situation where you have a number of web services to test but no one is able to provide full working examples of each API call? WStalker is a work aid to help developers / functional testers record API traffic to help facilitate security assessments by security testers and other tooling.
Project Ava: On the Matter of Using Machine Learning for Web Application Security Testing – Part 7: Development of Prototype #3 – Adventures in Anomaly Detection
In last week’s blog, our research team set out the process of creating a SQLi proof of concept. Overview In our previous prototypes we focused on text processing (vectorizing, word2vect, neural networks, etc.). We recognized that despite some signs of potential, the overall approach is difficult because: It’s not the way the human brain works […]