Personalized and Adaptive Virtual Reality/Augmented Reality - - PowerPoint PPT Presentation

Next Steps to Creating Personalized and Adaptive Virtual Reality/Augmented Reality Applications

Wendy Nilsen Computer and Information Science & Engineering Directorate National Science Foundation

National Science Foundation’s Mission

“To promote the progress of science; to advance the national health, prosperity, and welfare; to secure the national defense...”

Image: NSF's current headquarters, 4201 Wilson Boulevard, Arlington, VA. Credit: National Science Foundation

National Science Foundation 4201 Wilson Boulevard Arlington, Virginia 22230 TEL: 703.292.5111 | FIRS: 800.877.8339 | TDD: 800.281.8749 February 2017

NATIONAL SCIENCE BOARD OFFICE

OFFICE OF INSPECTOR GENERAL (OIG)

NATIONAL SCIENCE FOUNDATION

DIRECTORATE FOR MATHEMATICAL & PHYSICAL SCIENCES (MPS)

DIRECTORATE FOR GEOSCIENCES (GEO)

OFFICE OF THE DIRECTOR

703.292.8000

France A. Córdova

Director

Vacant

Deputy Director

NATIONAL SCIENCE BOARD (NSB)

703.292.7000

Maria T. Zuber

Chair

Diane L. Souvaine

Vice Chair DIRECTORATE FOR EDUCATION & HUMAN RESOURCES (EHR)

DIRECTORATE FOR COMPUTER & INFORMATION SCIENCE & ENGINEERING (CISE)

DIRECTORATE FOR BIOLOGICAL SCIENCES (BIO)

DIRECTORATE FOR ENGINEERING (ENG)

OFFICE OF INTEGRATIVE ACTIVITIES (OIA)

OFFICE OF LEGISLATIVE & PUBLIC AFFAIRS (OLPA)

OFFICE OF INTERNATIONAL SCIENCE & ENGINEERING (OISE)

OFFICE OF DIVERSITY & INCLUSION (ODI)

OFFICE OF THE GENERAL COUNSEL (OGC)

OFFICE OF INFORMATION & RESOURCE MANAGEMENT (OIRM)

OFFICE OF BUDGET, FINANCE, & AWARD MANAGEMENT (BFA)

DIRECTORATE FOR SOCIAL, BEHAVIORAL, & ECONOMIC SCIENCES (SBE)

Joan Ferrini-Mundy

Chief Operating Offic e r

NSF Supports All of Science & Engineering

CISE Divisions

Computer and Network Systems invent new computing and networking technologies and finds new ways to make use of current technologies. Information and Intelligence Systems studies the interrelated roles of people, computers, and information to increase our ability to understand data, as well as to mimic the hallmarks of intelligence in computational systems Office of Advanced Cyberinfrastructure supports and coordinates the development, acquisition, and provision of state-of-the-art cyberinfrastructure resources, tools and services essential to the advancement and transformation of science and engineering. Computing and Communication Foundations advances computing and communication theory, algorithms for computer and computational sciences and architecture and design of computers and software.

$934 M

FY 2016 research budget

8,301

proposals

1,908

awards

18,241

people supported

7,228

senior researchers

1,238

• ther professionals

490

postdoctoral associates

6,565

graduate students

2,660

undergraduate students

CISE by the Numbers: FY 2016

23%

success rate

372

panels

Data Current as of 2014

Computer Science Education Manufacturing, Robotics, & Smart Systems Big Data R&D

Image Credit: CCC and SIGACT CATCS

National Strategic Computing Initiative

Image Credit: Texas Advanced Computing Center Image Credit: Eliza Grinnell/Harvard SEAS

Understanding the Brain

Image Credit: ThinkStock Image Credit: Calvin Lin, University of Texas, Austin

Smart Cities

Image Credit: US Ignite

Cybersecurity

Image Credit: ThinkStock

For a comprehensive list of CISE funding opportunities, visit: http://www.nsf.gov/funding/pgm_list.jsp?org=CISE

CISE programs address national priorities

CISE’s Economic and Societal Context

• CISE is at the center of an ongoing societal

transformation and will be for decades to come.

• Advances in computing, communications and

information technologies, and cyberinfrastructure:

• accelerate the pace of discovery and innovation; and
• are crucial to achieving national and societal

priorities.

Risk & Resilience Health & Wellbeing Broadband & Universal Connectivity Food, Energy, & Water Systems Secure and Trustworthy Cyberspace

Image Credit: ThinkStock Image Credit: Wikicommons Image Credit: Nicolle Fuller, NSF Image Credit: Texas A&M University Image Credit: Public domain

Economic impact of CISE: From Federally- funded research to billion dollar industries

Advances in computing, communications and information technologies, and cyberinfrastructure:

• drive U.S. competiveness, (e.g.,

IT accounts for 25% of economic growth since 1995), and

• have profound impacts on our

daily lives.

From Continuing Innovation in Information Technology, NRC, 2012.

1970 1980 1990 2000 2010

University Industry Product ($B,$10B)

Virtual Reality/Augmented Reality Visioning Workshop – July 2017

• Workshop with 40+ academic, industry

and government experts to identify the scientific gaps in creating personalized and adaptive VR/AR systems

Workshop Goals

• To identify the research needed to

achieve individual personalization and adaptation in the areas of visual, auditory and tactile perception, as well as the social, behavioral and cognitive patterns key to adaptation

– Perceptual Systems – Social, behavioral and cognitive patterns

USE CASE: Training for First Responders

Jessie manages training and professional development for first responders. Thanks to new VR technologies, it has become relatively easy for Jessie to port volumetric models of real world settings into VR to build new training scenarios. Also, she can tweak the scenario settings for different training objectives. With earlier training systems, Jessie used to be frustrated because she couldn’t customize training scenarios for the trainees. This doesn’t result in effective training because her students enter the program with a wide variety of skill sets and

• experience. Specifically, she sees recruits employ different strategies -

both effective and ineffective - when navigating complex, rapidly changing environments and would like a training solution that supports such nuances.

User-Centric Hierarchical Benchmarks

• Hardware and software agnostic benchmarks should be developed for

assessing the end-to-end user experience to determine how well systems support a user to complete a given task. Categories to include: – Detection – Navigation (locomotion) – Selection – Manipulation

• Sample tasks to be included:

– Visual acuity – Contrast sensitivity – Disparity acuity – Localization in space (visual and auditory) – 3D motion acuity – Hand tracking – Distance estimation – Pointing – Interception

Optimizing the human-machine interface

Optimization should have the goal of a natural interaction - interaction that extracts a low cognitive load and a high level of mastery, which is adapted and personalized for users and tasks. Optimization should also achieve all this and fade into the human’s background.

• Optimization includes:

– Different types of user interfaces – Different kinds of interaction tasks – Match between task and setting

• Operation in the real world requires:

– Intelligent sensing/modeling of the environment – Algorithms for virtual objects in the physical space; – Methods to specify flexible environments that adapt to the physical world; – Studies of how the physical environments result in different user experiences and acceptance.

• Task-specific optimization

– Identify aspects that foster learning

• What are the universal aspects and what requires personalization?

Identifying User States & Traits

Understanding the emotional and physiological states of users real-time is critical to developing and assessing adaptation and ensuring VR/AR usability.

• Potential states and traits include:

– Engagement, which can range from mild interest to the full concentration “flow” – Presence “the sense of “being there” – Frustration – Boredom – Confidence or self-efficacy – Tenacity

• Internal states to detect/predict motion sickness

Identifying User Physical States

Future systems will need some form of body tracking to create lifelike avatars that mimic user characteristics.

• Tracking to include:

– Data to generate a motion model of a person – Data for accurate 3D visual and spatial audio

• What new analytics and models are needed to

generate real-time head related transfer functions?

• How do we personalize to the individual visual

system?

– Interpupillary distance – Acuity – Motion sensitivity

Dynamic Content Generation

Traditional content design either delivers a “one size fits all”

• solution. How can we create systems that dynamically adjust?
• Knowledge needed for understanding the effect of dynamic

changes

– Many solutions to optimize user comfort, adapt content dynamics to lower optic flow rate, change acceleration dynamics or content

• density. However, these adaptations have potential negative

effects on presence, immersion and engagement.

• Are there an adaptation management rule sets or logic that

can be developed?

• How much agency should the user have over adaptive system

processes?

• How much should the system communicate its logic?
• Finally, how can adaptation be assessed for its effectiveness,

desirability and perceived optimization?

Cybersickness

People vary quite widely in the three variables

• f cybersickness:

– Sensitivity – Adaptivity – Decay rate

• But how do these factors combine?
• To what extent these are state vs trait

variables?

• How are these variables affected by the user

context, environment and/or task?

Spatial Cognition

Navigation systems that understand user’s spatial knowledge and that adapt the amount of navigation help they give accordingly are needed.

• Needed information:

– Assess and utilize an individual’s knowledge and adapt to personalize interaction, navigation and locomotion – The effect of varying cues or adaptive geometry for varying speeds – The effect of locomotion speed-aware landmarks – Teleportation as a tool to allow for rapid, precise movement AND maintenance/development of spatial knowledge WITHOUT inducing discomfort – Detection of navigation/spatial problems or confusion

• In group settings:

– Affording multi-user calibration or coordination across people – Personalization for physically/virtually co-located users

• Are different interaction techniques or spatial layout necessary for

different users?

Social and Behavioral Factors

Current virtual environments rely on developers, users, and groups to structure and mediate social interactions. There is little knowledge about the impact of social behavior on VR/AR and the impact of VR/AR on social. How can adaptive systems support these functions by gauging an individual’s social intent and behaviors to deliver experiences that are more inclusive, safe, and comfortable?

• Topics include:

– Anonymity – Authenticity – Privacy and security – Legality of behaviors – Malleability of appearance – What is healthy social behavior in VR/AR? – Public vs Private control

Next Steps

• Personalization in VR/AR has a rich intellectual agenda –

highly creative, highly interactive, with enormous possibilities for changing the world!

• We have a thriving basic research community is the

foundation for long-term discovery and innovation to realize next generation VR/AR.

• VR/AR has the potential to transform a vast array of

tasks from high skill to education and entertainment.

CISE Partnerships

Partnerships build capacity, leverage resources, increase the speed of translation from discovery to innovation

societal org’s Federal agencies industry universities local gov’t international

Prescription 3: Regain America’s Standing as an Innovation Leader by Establishing a More Robust National Government-University- Industry Research Partnership

Stay Connected

• Subscribe to get NSF updates by

email at www.nsf.gov.

• Subscribe to receive special CISE

announcements: – Send a message to: join-cise- announce@lists.nsf.gov with no text in the subject or message body.

• Visit the CISE website often:

http://www.nsf.gov/CISE

• Talk to Program Directors:

http://www.nsf.gov/staff/staff_list.jsp?

• rg=CISE&from_org=CISE.
• Follow CISE on Twitter @NSF_CISE.

Get NSF Updates by Email

Thank you!

Wendy Nilsen wnilsen@nsf.gov 703-292-2568