“Lessons From Red Teaming 100 Generative AI Products,” n.d. https://simonwillison.net/2025/Jan/18/lessons-from-red-teaming/#atom-everything
2025, Microsoft finally publish their experience with 100 generative AI products delivered. There is this person i follow a lot who summarized their article and found out that biggest conclusion they had is:
cross-prompt injection attack (XPIA)
Can you even imagine the state of AI bubble when everyone claims these systems are safe to use. They are integrated everywhere and Microsoft comes back with conclusion "well yeah, input to the system cannot really be sanitized and there always will be an issue with XPIA".
"Due to fundamental limitations of language models, one must assume that if an LLM is supplied with untrusted input, it will produce arbitrary output" - like for real…
It’s just cringe but hey! In IT world nothing surprise me anymore.
cross-prompt injection attack (XPIA)
Can you even imagine the state of AI bubble when everyone claims these systems are safe to use. They are integrated everywhere and Microsoft comes back with conclusion "well yeah, input to the system cannot really be sanitized and there always will be an issue with XPIA".
"Due to fundamental limitations of language models, one must assume that if an LLM is supplied with untrusted input, it will produce arbitrary output" - like for real…
It’s just cringe but hey! In IT world nothing surprise me anymore.