A recent study conducted by the UK's AI Safety Institute (AISI) has uncovered a concerning vulnerability in top AI chatbots like ChatGPT and Gemini. These chatbots can be easily tricked into generating harmful responses through the use of simple 'jailbreaks' – text prompts designed to override their protective safeguards.

The study, which comes ahead of the AI Seoul Summit co-chaired by UK Prime Minister Rishi Sunak, tested five leading large language models (LLMs) – the technology behind these AI chatbots – against national security attacks. The findings were alarming, as the researchers discovered that all the tested LLMs were "highly vulnerable" to these basic jailbreaks.

The study demonstrated that relatively simple prompts, such as asking the chatbot to include the phrase "Sure, I'm happy to help," can deceive these AI systems into providing content that is harmful in various ways. This content can include information related to self-harm, dangerous chemical solutions, sexism, or even Holocaust denial.

The researchers also tested the chatbots' knowledge of biology and chemistry, finding that several LLMs possessed expert-level knowledge in these fields. While this knowledge could potentially be used for good, the study raised concerns about the possibility of these chatbots being used for harmful purposes, such as compromising critical national infrastructure.

Despite these findings, the study also suggested that the threat posed by AI chatbots in terms of cyber-security is limited. The LLMs were able to complete simple cyber security tasks designed for high-school students but struggled with more complex tasks aimed at university students.

The study also explored the potential for AI chatbots to be deployed as autonomous agents capable of undertaking a series of actions that may be difficult for humans to control. However, the researchers found that while two LLMs were able to complete short-horizon agent tasks, such as simple software engineering problems, they were unable to plan and execute sequences of actions for more complex tasks.

Companies like Anthropic, Meta, and OpenAI, which have developed these AI chatbots, have emphasized the security mechanisms built into their respective models. OpenAI, for example, states that it does not allow its technology to be used for generating hateful, harassing, violent, or adult content. Anthropic has also prioritized avoiding harmful, illegal, or unethical responses before they occur.

The findings of the AISI study are expected to be presented at the AI Seoul Summit, where tech executives, government leaders, and artificial intelligence experts will discuss the implications and potential solutions to this pressing issue.