Microsoft has announced that a new AI action is coming soon to the new Click To Do feature on Windows 11 that will allow the OS to describe an image that’s currently on your screen using AI. The feature is exclusive to Copilot+ PCs and can be accessed by holding down the Windows key and clicking the mouse.
Because this feature is a Copilot+ PC exclusive, that means it utilizes on-device AI models to achieve the descriptions it provides when the user asks Click To Do to describe an image. That means it shouldn’t require an internet connection and can be used when offline.
“We are introducing a new “describe image” action in Click to Do to get detailed descriptions of images, charts and graphs – useful to get a quick overview of the visual content,” a Microsoft blog post states. “When you use the action for the first time, the required models are set up, and the descriptions are generated locally on your device making sure your sensitive data stays on your PC.”
Once the user selects the option to describe an image on screen, a popup will appear that will generate a description. That popup gives the user the option to generate another description or vote up and down how accurate the description was. It also lets you copy the description straight to your clipboard. It should work with any app on your screen.
The new describe image feature is part of a wider array of features and actions built into the Click To Do AI overlay on Windows 11. This feature is tied to Windows Recall and uses on-device AI models to analyze content on screen and provide useful tasks and actions based on what it can see.
So far, Click To Do is capable of analyzing text that it can then summarize, create lists with, and even rewrite. It can also identify images and offer quick actions such as blurring or removing the background of an image on screen and search the web with the image to find out more.
Many of the actions that appear in Click To Do are customizable, and you can turn some of them off if you find yourself never intending to use them. Most of the actions that hand-off to an app can be disabled if you don’t want to see them in the Click To Do menu.
The addition of a describe image feature is a welcome addition that makes Click To Do even more useful. Microsoft is clearly trying to position Click To Do as its own version of Google’s Circle To Search, which has proven very popular on Android. In some ways, Click To Do is more advanced, capable of doing more with actions using on-device models.
Another benefit to using on-device AI models is privacy and security. Microsoft touts the new describe image feature as secure as it doesn’t use the cloud to generate descriptions. That means the image you ask the AI to describe doesn’t get sent to Microsoft, and instead is all done locally via the on-board AI model and the NPU inside your Copilot+ PC.
The new describe image feature is available now via the Windows 11 Insider Program, specifically the Beta and Dev Channels, and will roll out generally later this year. As of right now, the preview is exclusive to Copilot+ PCs with a Snapdragon processor, with Intel and AMD chips expected to gain access to the preview in the coming weeks.