Microsoft 'Seeing AI' app is helping the blind

A Microsoft developer has designed and created an app that helps the visually impaired by describing what is happening right in front of them at the touch of a button- well sort of.

This app can be used via smart glasses from Pivot head which are worn by the user. When they need a helping hand- so to speak- they swipe the side of the glasses in order to take a snapshot of what is in front of them. This could be a crowded room, a young man jumping on a skateboard or a person running. The glasses then verbally describe what is in front of the wearer, giving the user a clear understanding of their surroundings. This is something that traditional aids such as a guide cane cannot provide.

The aim of the project was to develop and deliver a product that helps those with limited to no eye function to be able to better understand what is going on around them.

This app is so smart it also uses facial recognition technology to gage how a person is feeling. For example the user takes a snapshot of their meeting room, the app then describes the general age and sex of the people in the room along with an adjective to describe the emotion on their face whether that is angry, sad, surprised etc.

A number of recent breakthroughs in computer vision has allowed this app to be possible- that and decades of research. Seeing AI is one of the latest advancements in computer vision incorporating years of research in numerous fields including natural language processing, machine learning and image recognition.

There are a number of apps on the market designed to aid and support the visually impaired however the feedback on these is that they can be hard to use and have small separate functions rather than an app that has multiple uses. Microsoft’s aim with Seeing AI was to make their products more accessible. This app gives the user more independence and allows for less reliance on others.

Take a look at the video here: