Task Aware Browsing: AI Assisted Web Access for Blind Users

A new approach for blind users to access web pages is presented. It uses machine learning and artificial intelligence (AI) to overcome problems with conventional browsers and screen readers, particularly on an unfamiliar web site. A set of typed commands is provided to allow a task-oriented approach to completing tasks on a web application. Rather than having to look for a search box, the user need only type "search," or its equivalent hotkey. Rather than tabbing through a page of clutter, only the results of the search are provided. It uses AI to convert the spatial layout of active controls and the subsequent results, to a linear dialog model. The new approach is called Task Aware Browsing, TAB for short. Work performed to date indicates that machine learning techniques will allow this approach to work on a significant number of web sites. User testing suggests that it provides an appreciable improvement in the speed with which blind users can accomplish desired tasks compared to existing screen readers. This design also allows for speech assistants to work on any site that TAB can work on, without the need of specific scripts for that site.