Anthropic publicly releases AI tool that can take over the user’s mouse cursor

AI software company Anthropic has announced a new tool that can take control of the user's mouse cursor and perform basic tasks on their computer.

Announced alongside other improvements to Anthropic's Claude and Haiku models, the tool is straightforwardly called "Computer Use." It's available exclusively with the company's mid-range 3.5 Sonnet model right now, via the API. Users can give multi-step instructions (Anthropic claims it can go for tens or even hundreds of steps) to accomplish tasks on the user's computer by "looking at a screen, moving a cursor, clicking buttons, and typing text."

Here's how Anthropic says it works:

When a developer tasks Claude with using a piece of computer software and gives it the necessary access, Claude looks at screenshots of what’s visible to the user, then counts how many pixels vertically or horizontally it needs to move a cursor ...