Computer Use Tool: Automating Web & Desktop Apps with Copilot Studio

Microsoft Copilot Studio is a cloud-based platform that empowers you to create custom AI agents (chatbots or assistants) for internal or external use, without requiring deep AI/ML expertise or coding skills. You can build these agents using plain language prompts or a low-code/no-code graphical interface.

These agents can connect to multiple data sources, corporate systems, APIs and external knowledge bases. Once created, they can be published across channels like Microsoft Teams, mobile apps, websites and more. Copilot Studio ensures enterprise-grade security, governance and scalability, making it ideal for organizations.

If you’re new to Copilot Studio, Microsoft offers an applied skill course to help you design and deploy custom agents.

Understanding Tools in Copilot Studio

Tools are the building blocks that allow your agent to interact with other systems. Each tool represents a specific capability, enabling your agent to perform tasks based on user input or triggers.

For example, you can add tools to:

• Send emails via Office 365 Outlook
• Fetch weather forecasts
• Read/write data from Dataverse
• Post messages to Microsoft Teams

Options for Adding Tools

Copilot Studio offers multiple ways to integrate tools into your agent:

• Prebuilt Connector: Access thousands of ready-to-use API connectors from Microsoft and third-party services.
• Custom Connector: Define custom connections using Power Platform Connectors for unique systems.
• Agent Flow: Create workflows with multiple tasks.
• Prompt: Use single-turn prompts powered by AI models.
• REST API Connection: Add API endpoints and methods as tools.
• Model Context Protocol (MCP): Connect to MCP servers for resource access.
• Computer Use (Preview): Automate interactions with desktop apps and websites via GUI navigation.

What is Computer Use in Copilot Studio?

The Computer Use tool (currently in preview) enables your agent to perform tasks on a Windows computer by simulating user actions like clicking buttons, navigating menus and entering text. This is particularly useful when APIs are unavailable, allowing automation through the graphical interface.

Imagine automating:

• Data extraction
• Invoice processing
• Automated data entry

If you’re interested in desktop automation, check out the course PL-500: Microsoft Power Automate RPA Developer Course to learn advanced robotic process automation techniques.

How Does It Work?

Computer Use is powered by Computer-Using Agents (CUA), an AI model combining reasoning and vision to interact with GUIs. It adapts to interface changes, ensuring smooth operation even when buttons or screens change.

You simply provide natural language instructions with no coding required.

Prerequisites:

• Available only in environments with the United States region selected.
• Ensure Generative Orchestrator is enabled for your agent.

Steps to Add Computer Use to Your Agent

Here’s how you can integrate Computer Use into your Copilot Studio agent:

1. Navigate to Tools → Add Tool.
2. Select New Tool → Computer Use.
3. Provide instructions for the task (templates are available).
4. Choose a Windows PC or a hosted browser for automation.
5. Configure fields:
   • Name: Display name for the tool.
   • Description: Purpose and usage context.
   • Instructions: Detailed steps, app names, URLs.
6. Review additional settings (Inputs, Credentials, Access Control).
7. Save changes.

Testing and Publishing

Testing is a critical step before publishing your agent:

• Click Test after saving the tool.
• View logs and actions on the left panel; preview execution on the right.
• Stop testing anytime using Stop Testing.

Agents can operate in two modes:

• Conversational: Interact with users via channels like Teams.
• Autonomous: Run tasks in the background without user input.

Computer Use is most beneficial for autonomous agents, enabling seamless background automation.

Next-Gen Automation with AI

The Computer Use tool in Microsoft Copilot Studio represents a significant leap forward in automation capabilities. By enabling agents to interact directly with graphical interfaces, organizations can automate tasks that were previously dependent on manual intervention or unavailable APIs. This flexibility opens doors for streamlining workflows, reducing operational costs and improving efficiency across departments.

About CloudThat