Eyes-free Computing

Past, Present And Future

T. V. Raman

Google

http://emacspeak.sf.net/raman

https://tvraman.github.io

October 25, 2018

Efficient Eyes-free Access

Goal AUI Multimodal Web Mobile Search Smart Assistant Conclusion

Eyes-free Computing

4 / 28

■

Information — When we want,

■

Information — Where we want,

■

Information — The way we want!

This Journey Began 30 Years Ago.

Milestones Along The Way

Goal AUI Multimodal Web Mobile Search Smart Assistant Conclusion

Eyes-free Computing

5 / 28

Electronic Information Is Display-Independent!

■

ASTER — Audio System For Technical Readings —

1990

■

Emacspeak — The Complete Audio Desktop —

1995

■

PDF — Capture and extract document structure —

1996

■

Web — Design standards for multimodal interaction —

2000

■

Search — Deliver results the way the user wants —

2005

■

Mobile — Ubiquitous information access —

2008

■

Google Assistant — eyes-free Interaction —

2014

■

The best is yet to come!

Audio Format And Browse Documents

Goal AUI Multimodal Web Mobile Search Smart Assistant Conclusion

Eyes-free Computing

7 / 28

ASTER — Audio System For Technical Readings

■

Extracted logical structure from (LA)TEX documents

■

Used Audio Formatting Language (AFL) to render result

■

Interactive browser enabled multiple audio views

Electronic Information Is Display-Independent!

Information Is Display Independent

Goal AUI Multimodal Web Mobile Search Smart Assistant Conclusion

Eyes-free Computing

8 / 28

Single Source, Multiple Outputs

Info

Visual

Audio

Speak

Emacspeak: Complete Audio Desktop

Goal AUI Multimodal Web Mobile Search Smart Assistant Conclusion

Eyes-free Computing

9 / 28

Speech-enabled applications

■

Speak information —not its visual presentation

■

Text-To-Speech leverages application context

■

Audio formatting makes communication efficient

■

Auditory icons enhance user experience

■

Speech-enable all aspects of user interaction

UI —Capture Intent, Grab Attention

Goal AUI Multimodal Web Mobile Search Smart Assistant Conclusion

Eyes-free Computing

10 / 28

Input

Output

data res

intent

attn

2000 — The Multimodal Web

Goal AUI Multimodal Web Mobile Search Smart Assistant Conclusion

Eyes-free Computing

12 / 28

The Web the way you want (1st Edition)

■

Limited forms of the Web were available on hand-helds

■

Devices of the day —Smart PDAs, network-enabled phones

■

Speech interaction was viewed as the next-frontier

Web: Separate application and Interaction Logic !

Opportunity: High-Level Semantics

Goal AUI Multimodal Web Mobile Search Smart Assistant Conclusion

Eyes-free Computing

13 / 28

■

Anywhere, Anytime access requires rich content semantics

■

Rich semantics is a pre-requisite for universal access

■

Mainstreams a core Accessibility requirement

Multimodal Web (1st Edition) was ahead of its time!

Being Digital

Goal AUI Multimodal Web Mobile Search Smart Assistant Conclusion

Eyes-free Computing

14 / 28

From Atoms To Bits

■

Physical world comes on-line —atoms change to bits

■

Bits —electronic information —is computable

■

Each atoms → Bits transition enables new functionality

■

Bits → Atoms enables new forms of interaction

Each transition is an opportunity to innovate!

Atoms → Bits → Atoms

Goal AUI Multimodal Web Mobile Search Smart Assistant Conclusion

Eyes-free Computing

15 / 28

Opens up new Access opportunities

Example

Opportunity

Digital Maps

Explore the world

Digital Maps

Tactile graphics

Electronic Books

Access world’s knowledge

Mathematical Content

Transform to multiple presentations

Scientific Diagrams

3D printing for exploration

Core Value Of Search

Goal AUI Multimodal Web Mobile Search Smart Assistant Conclusion

Eyes-free Computing

17 / 28

Effective, timely access to relevant information!

Effective

Ensure rapid task completion.

Timely

Deliver results as quickly as possible.

Relevant

Interpret query, and rank results.

Effective Information Access

Goal AUI Multimodal Web Mobile Search Smart Assistant Conclusion

Eyes-free Computing

18 / 28

Metric: Task Completion

■

Interpret user intent,

■

Retrieve relevant information,

■

Present it effectively to enable task completion.

Mobile Access

Goal AUI Multimodal Web Mobile Search Smart Assistant Conclusion

Eyes-free Computing

19 / 28

Enables true ubiquitous access (2008)

■

All the world’s information in the cloud,

■

Computable and transformable,

■

Viewable via a multiplicity of clients.

Android and ChromeOS

UI Defines The Device

Goal AUI Multimodal Web Mobile Search Smart Assistant Conclusion

Eyes-free Computing

21 / 28

UI Peripherals determine size and shape of devices!

■

Keyboards, Mice and Monitors defined the size and shape of computing.

■

Smart phones turned these into flat shiny piece of glass.

■

Smart speakers make computers disappear.

■

Smart displays are information appliances.

Smart Assistant

Goal AUI Multimodal Web Mobile Search Smart Assistant Conclusion

Eyes-free Computing

22 / 28

■

Enable ubiquitous hands-free, eyes-free information access.

■

Speech interaction is a first-class citizen.

■

Enable contextual in-the-moment assistance!

User-Aware Interfaces

Goal AUI Multimodal Web Mobile Search Smart Assistant Conclusion

Eyes-free Computing

23 / 28

UI: Determine User Intent, Grab User Attention

■

User-Aware Interfaces —eliminate friction in the UI.

■

User-Awareness ↓

◆

User location, e.g., home, work,

◆

Usage context, e.g., driving,

◆

User abilities, e.g., hands-busy,

◆

UI optimized to the user’s current context.

Ubiquitous Assistant

Goal AUI Multimodal Web Mobile Search Smart Assistant Conclusion

Eyes-free Computing

24 / 28

■

Home appliances provide contextual assistance

■

Smart home controlled via multiple interaction modalities

■

Smart wearables that sense environment

■

Smart wearables that effectively grab user attention

■

Single Assistant, multiple access paths

Assistant maintains context across devices

Impact On Universal Access

Goal AUI Multimodal Web Mobile Search Smart Assistant Conclusion

Eyes-free Computing

25 / 28

■

AI-powered Assistant is aware of user needs

■

User-aware interfaces sense user context

■

Control connected devices via a multiplicity of interfaces

■

Deliver interface to match user’s abilities

Conclusion

Goal AUI Multimodal Web Mobile Search Smart Assistant Conclusion

Eyes-free Computing

27 / 28

■

Electronic information is display-independent

■

UI determines size and shape of computing

■

Ubiquitous information access requires User Aware UI

Document Outline