AI agents are now performing online tasks, bringing new risks

How AI agents interact with web pages

Until recently, asking an AI to search for a flight ticket would only provide information about prices and schedules. The next step still required user intervention: opening the website, selecting the flight, entering details, and completing the purchase. Now, AI agents are changing this task division, enabling technology to interact directly with web pages.

OpenAI has developed the Computer-Using Agent (CUA), a technology that can interpret on-screen content and use a virtual mouse and keyboard to click, scroll, and type. Initially tested in Operator, this feature is now part of ChatGPT’s agent mode capabilities.

Unlike traditional chatbots that generate responses, agents can execute a sequence of actions to achieve a result. In a browser, this involves identifying a button, filling in a field, opening another page, comparing information, and continuing the process. The technology monitors the interface through screenshots, interprets the current page state, and decides the next action. If something changes during execution, the agent can attempt to adapt its plan.

Tasks that can be automated

This technology is useful for multi-step activities. OpenAI provides examples such as form filling, product searches, organizing information across different pages, and performing administrative tasks.

In ChatGPT, the agent mode combines visual navigation with other tools like web search, terminal, and connectors. A task can begin with a search, proceed with information analysis, and conclude with an action on a website.

Current risks and limitations

Granting AI the ability to act also introduces new risks. A webpage may contain malicious instructions, an agent may misinterpret an action, or a website may block automation.

As a result, current systems impose limits on certain actions. For instance, ChatGPT’s cloud browser pauses when it requires information, login, or user confirmation. OpenAI also advises users to verify the website address, proposed steps, and confirmation requests, and to avoid entering passwords, security codes, or payment information directly into the conversation.

The change, therefore, goes beyond the ability to click. The browser becomes an environment where AI can perceive, decide, and act, while the user defines the goal and monitors steps that may have significant consequences. For those working with digital tools, understanding this new dynamic is becoming increasingly important.