Source: Anthropic, Introducing Computer Use (Claude 3.5 Sonnet)

Anthropic's October 2024 announcement of computer use in public beta, with early OSWorld results and stated limitations.

Contents

Summary

First frontier-model computer use in public beta: Claude views screenshots and moves the cursor, clicks and types through an API tool.

Key claims

  • Claude 3.5 Sonnet scored 14.9% on OSWorld screenshot-only (next best 7.8%), 22.0% with more steps.
  • Described as experimental and at times cumbersome and error-prone; scrolling, dragging and zooming were hard.
  • Developers advised to start with low-risk tasks.
  • New classifiers detect computer use and potential harms (spam, misinformation, fraud).

Feeds

Computer-use agents

Suggest changes →