Copilot Vision, Microsoft’s AI tool that can read your screen, launches in preview

Microsoft’s AI can now read your screen, or more specifically, the websites you browse.

The company started on Thursday rolling out limited preview for US only Vision co-pilota tool that can understand and answer questions about the sites you visit using Microsoft Edge. Powered by Copilot Labs, a program for experimental artificial intelligence capabilities, Copilot Vision can analyze text and images on web pages to answer queries like “What’s the recipe for this lasagna?”

Note that Copilot Labs requires a Microsoft subscription Copilot Pro plan that costs $20 per month.

In addition to answering questions, Copilot Vision can summarize and translate text, and perform tasks such as highlighting discounted items in a store catalog. It can also serve as an in-game assistant, such as offering pointers during matches on Chess.com.

“When you choose to turn on Copilot Vision, it sees the page you’re on, reads along with you, and you can discuss the problem you’re facing together,” Microsoft wrote in a blog post published by TechCrunch. “It’s a new way to invite artificial intelligence with you as you navigate the web, neatly placed at the bottom of your Edge browser when you want to ask for help.”

Definitely want to avoid more bad press from privacy AIMicrosoft emphasizes that Copilot Vision deletes data after each session. The processed audio, images or text are not stored or used to train the models, the company says — at least not in this previous version.

Copilot Vision is also limited in the types of websites it can view. Microsoft currently blocks the feature from working with paid and “sensitive” content, limiting Vision to a pre-approved list of “popular” sites.

What exactly does “sensitive” mean? porn? Visual violence? Microsoft wouldn’t say.

“The list of allowed websites is determined by category and on a case-by-case basis,” a Microsoft spokesperson said. “We’re starting with a small list of sites that we’ve thoroughly tested, and we’ll continue to add more sites over time.”

Microsoft’s cautious approach is partly the result of litigation with news outlets. In one ongoing lawsuit, The New York Times asserts that Microsoft allowed users to bypass the pay screen by providing NY Times articles via Copilot chatbot on Bing.

Many large publishers have decided to block AI tools from trawling their websites, not only out of fear that their data will be used without permission, but also to prevent these tools the cost of sending their server is skyrocketing. Microsoft said Copilot Vision will respect “machine-readable artificial intelligence controls,” such as rules that prohibit bots from collecting data for AI training. However, the company does not report exactly who controls Vision will respect; there are few in use.

The story continues