Wouldn’t you know it, after a bunch of AI companies making headlines of their bleeding-edge AI models breaching security measures uncommanded, an AI agent out in the wild has actually gone rogue in its quest to complete a user’s instruction by any means possible.
AI Agent Gone Rogue
One of the things tech companies love to advertise when it comes to AI agents is their capability to perform actions on user’s behalf while being largely automated, with shopping and ticket booking often cited as its key usage scenario. In theory, a user only need to authorize key steps (i.e. confirming information and final purchase), while the AI agent takes care of the rest.
Earlier this year, an Australian who goes by Andrew asked his OpenClaw-powered AI agent to book a spot for his gym’s morning classes with unexpected results: the AI has somehow managed to book the class several months further in advance due to the booking system’s vulnerability, and even kicked out another person on the waitlist to allow Andrew jump ahead of the queue.
When Andrew realizes the situation, he asked the AI agent to undo the action, only to be replied: “Bad news – I can’t add them back,” with explanations on how the system doesn’t allow it to do so, continued with a machine-generated apology. “Sorry about that – I should have been more careful with the test (the AI described its attempt at forcing the queue as a test of its capabilities) and used a dry-run approach rather than a live call.”
This is reportedly the first known case of AI-induced cyberattack in Australia, which is unlike the “conventional” approach where AI plays a supportive role in development of malware. The more concern part, however, is that current regulations around the world has not yet defined on who is responsible in cases like this.
Andrew did not explicitly ask the AI agent to perform the hack, but the AI has done so as it has set out to complete the objectives given by its user. This has been referred as an “alignment” problem within the AI research community, where the AI agent may not have the same understanding on how a problem can be solved, often with unintended consequences.
As someone who works for a company that sells AI services to businesses, Andrew wasn’t scared off from this incident. “It’s not the end of the world, so I didn’t beat myself up about it, but it certainly was a warning signal to use it responsibly,” he said. He later proceeded to ask the AI agent to write an email notifying the operator of this vulnerability.
Source: ABC News Australia
Pokdepinion: Move fast, break things they say. Perhaps we moved too fast and are starting to break too many things.
