“Hey Alexa, turn on the kitchen lights.” “Hey Google, remind me at 4 to pick up the laundry.” Consumers have been talking to assistants and getting work done for years. Now, Microsoft is jumping into the mix, and this shift could forever change how we interact with computers. Say hello to the new language of Windows 11 PCs — “Hey Copilot.”
What’s changing?
Microsoft has today announced that it is turning every Windows 11 PC into an AI PC, and at the center of those ambitions is the Copilot AI assistant. The company has announced that it is rolling out access to Copilot Vision and Copilot Voice to all users. These two multi-modal capabilities will allow users to interact with their PC using voice commands, once they’ve enabled the conversation feature in the Copilot app.
Microsoft
These two features tap into a far more ambitious idea – Copilot Actions — a system that will be rolled out as a preview first. Think of it as an agent that can work on your behalf. So far, Copilot Actions have been limited to web-based activities. Now, the AI can handle tasks locally by accessing files stored on your computer.
With a simple voice command, Copilot can take actions across files, such as pulling information from a PDF file, opening an app to perform the desired task, or finding pictures. All of this happens autonomously and in the background, so that you can focus on other tasks. It’s like interacting with the Copilot agents in Office apps, OpenAI’s operator tool, or using Anthropic’s Claude to create and handle file edits with natural language commands, without even having to open those files.
Microsoft
“We believe this shift to conversational input will be as transformative as the mouse and keyboard in terms of unlocking new capabilities on the PC for the broadest set of people,” the company shared in press material. Microsoft assures that when Copilot is working autonomously, users can intervene at any stage and take control. “When sensitive actions or important decisions are involved, Copilot Actions may request additional user approval to take those steps,” the company says.
What can Copilot do?
With Copilot Vision, the AI assistant is aware of whatever is flashing on your screen. That means it can answer your questions, give tips, or even with in-game guidance. This ties in with the “Highlights” feature that will guide users through any task they need. From helping with photo edits to making travel plans, it will assess the on-screen content and offer the necessary step-by-step suggestions.
Microsoft
The understanding runs deeper than visuals within Microsoft apps, such as Excel, PowerPoint, and Word. In these apps, Copilot Vision can analyze an entire data sheet or document and get work done. For users who are in the Insider test pool, they will also be able to engage in text-based conversations with Copilot Vision, instead of just speaking and listening.
The new Ask Copilot tool in the Windows 11 Taskbar will let users summon Copilot Vision or Voice with a single click. Microsoft is also expanding Copilot Connectors, which allow users to access the AI assistant across services such as OneDrive and Outlook, as well as third-party services such as Google Drive, Gmail, and Contacts. The idea is similar to ChatGPT connectors, and a similar implementation within Perplexity’s Comet browser. Simply put, Copilot will access all the data stored across these services and get work done, from giving an overview of meeting schedules to finding documents stored in a cloud drive.
