A New Approach to Developing Computer Use Agents

Researchers at Google Cloud and DeepMind have collaborated to create a groundbreaking framework aimed at revolutionizing the development of computer use agents (CUAs). The main focus of this innovative framework, known as Watch & Learn (W&L), is to tackle the challenge of acquiring high-quality training examples for CUAs on a large scale.

Traditional methods of gathering training data for CUAs often involve human annotation, which can be time-consuming and costly. However, W&L offers a solution by automatically extracting demonstrations from raw videos, eliminating the need for manual annotation.

The Significance of Training Data in CUA Development

Utilizing the vast resources of video tutorials and screencasts available online, CUAs can acquire valuable domain knowledge and learn how to navigate various applications. The challenge lies in transforming these videos into annotated trajectories, a process that can be inefficient when done manually.

Previous approaches to address this data bottleneck have proven to be ineffective, often resulting in low-quality training examples. However, the Watch & Learn framework takes a different approach by reframing the problem as an “inverse dynamics objective,” making it easier to learn and generalize across applications.

The Watch & Learn Framework in Action

The Watch & Learn framework consists of three key stages: training an inverse dynamics model (IDM), retrieving raw videos, and training CUA agents. By utilizing agents to interact with live web pages, the researchers were able to create a large corpus of state transitions, which were then used to train the IDM.

The IDM, a small transformer model, outperformed existing foundation models in predicting transition actions. By retrieving videos from platforms like YouTube and running them through the IDM, high-quality trajectories with accurate action labels were generated.

These trajectories not only serve as training examples for CUAs but also enhance the performance of CUAs on bespoke tasks through in-context learning. By incorporating additional reasoning annotations into the observation/action examples, CUAs can improve their performance without the need for costly specialized models.

Experimental Results and Implications

Experiments conducted with closed and open-source models on the OSWorld benchmark demonstrated the effectiveness of the Watch & Learn framework. Fine-tuned open-source models and general-purpose multimodal models showed improvements in performance, showcasing the scalability and practicality of web-scale human workflows in advancing CUAs.

With the potential to turn existing video corpora into valuable training data for CUAs, enterprises can leverage Watch & Learn to enhance their computer use agents without the need for manual annotation. As technology continues to evolve, the field of CUA development is poised for significant progress and innovation.