Developing Immersive Applications: Evaluating Immersive Experiences
Main image
Intro
This video will discuss common ways to evaluate user experiences for immersive systems.
The key topics to be covered in the discussion of evaluating user experiences for immersive systems will be addressed.
Evaluating user experiences is a crucial aspect of developing immersive applications, specifically for immersive systems.
The Outline
To develop immersive applications, it's essential to understand the motivation behind learning how to evaluate such systems before diving into technical implementation concepts.
Describing immersion can be done in two ways: as system properties and as user experiences, with the latter involving a more detailed look at experiential constructs.
Three popular experiential constructs that will be explored in more detail are presence, flow, and cyber sickness.
The concept of audiences will be used to relate these concepts to design, providing a framework for learning and application.
When learning a new programming framework, it's common to get caught up in learning new features one by one, but this approach often neglects evaluation and testing until the end.
Traditional courses often follow a similar structure, covering technical skills and concepts first and leaving evaluation and testing until later.
Evaluation-first approach
The primary focus when developing immersive applications should be on creating a desired immersive user experience, which is what truly matters to the users, rather than just the technical concepts involved.
Understanding that the end product is for the users, not just for personal satisfaction, is crucial, and acquiring knowledge on how to evaluate the quality of the immersive application at an early stage is important.
The concept of test-driven development (TDD) is relevant, where test cases are created before coding features, and this approach is applied to evaluating immersive applications by learning how to properly evaluate user experiences first and then building up knowledge on how to create such experiences.
In this approach, the evaluation of user experiences of immersive applications is done first, and then knowledge on how to create such experiences with different technical tools and hardware is built up cumulatively.
The goal is to understand that the technical concepts are a means to an end, and the end desired immersive user experience is what really matters to the users.
Acquiring a good level of knowledge on how to evaluate the quality of the immersive application at an early stage is essential for creating a successful end product.
The evaluation-first approach is emphasized, where the evaluation of the immersive application is done at the start, rather than at the end, similar to the TDD approach.
Immersive system
An immersive system is the type of system being built, and it's essential to understand its characteristics and differences from other systems.
Designing for immersion is distinct from designing for other user requirements, and this difference should be considered when developing applications.
Immersion can be thought of as a quality or property of any application, and developers should decide how immersive they want their application to be during the design process.
The concept of immersion is crucial to understand, and developers need a clear idea of what immersion entails when designing an application.
The level of immersion required can vary depending on the type of application being developed, such as an Interactive Learning simulation versus a utility application like a to-do app.
What is immersion
Immersion can be measured to determine different levels of immersion, with the dictionary definition providing a starting point.
The dictionary definition of immersion is a deep mental involvement in something, which gives a general direction for understanding the concept.
In the context of building systems, immersion can be thought of in two ways: as system properties and as user experiences.
As system properties, immersion refers to the characteristics of a system that contribute to an immersive experience.
As user experiences, immersion is related to the user's perception and engagement with the system.
Immersion as system properties
The system properties that produce immersive experiences include technical specifications such as resolution, field of view, and sensors, which can be found on the specs sheet of devices like the Meta Quest Pro.
The Meta Quest Pro's specs sheet lists features like 1800 x 1920 pixel resolution per eye, separate LCDs with local dimming, 106 degrees horizontal and 96 degrees vertical field of view, and stereoscopic color pass-through.
Other technical properties of the Meta Quest Pro include state-of-the-art eye phase and motion sensors, a wide range of interpupillary distance adjustment mechanics, spatial audio, and a three-mic array.
These technical qualities of the system facilitate the feeling of immersion and can be found on both the hardware and software sides of the system.
On the software side, examples of system properties that contribute to immersion include visual fidelity of graphics, realism of AI-driven human characters, interaction controls, and natural controls like finger tracking.
The description of these system properties is a way to describe the user experience of immersion.
Immersion as user experiences
Immersion is a complex concept that combines psychological and physiological constructs, allowing for the measurement of desired user experiences and informing the design of immersive applications.
Understanding immersion as an experience is an ongoing academic pursuit in the field of Human-Computer Interaction (HCI).
The main focus of research in this field includes three popular constructs related to immersion: presence, flow, and cyber sickness.
These three constructs are not exhaustive, but rather a few examples that can be used to determine how well an application provides an immersive experience to users.
The concept of presence is one of the key constructs related to immersion and will be explored further.
Presence
Presence is a key defining experience of VR applications and immersive applications, representing the feeling of being transported to an alternate space.
The concept of presence has been interpreted in many ways since it was coined by Minsky in 1980, but all interpretations converge to the notion of how much one feels like being in a different space.
Presence can be broken down into various dimensions, including perceptions of environmental interaction, perceived fidelity and realism of the simulated environment, and the degree to which these are conveyed through our senses.
Presence can be differentiated between physical presence, the sense of being physically relocated to the virtual space, and social presence, the sense of being around other virtual beings.
Individual differences, such as personality traits like openness and extroversion, can modulate the feeling of presence, with some people experiencing higher presence than others.
Presence can be measured using subjective data, objective data, or a mix of both, with subjective measures like self-reporting questionnaires and interviews being commonly used.
The Igroup Presence Questionnaire (IPQ) is a popular, validated self-reporting questionnaire that segregates presence into three parts: spatial presence, involvement, and realness.
Spatial presence refers to the feeling of physical existence in the virtual space, involvement is how captivated one is in the virtual world, and realness is how real the virtual world feels compared to the actual world.
CHI VR locomotion paper
A VR commuting Simulator was developed for the Land Transport Authority of Singapore, and this prior study will be used as a running example throughout the video to discuss measuring experiential constructs.
The study involved 42 participants and plotted the average scores of the subdimensions of the IPQ across different ways users performed locomotion.
Although the differences in the study were not significant, it demonstrates how instruments like the IPQ can inform application design and how different implementation approaches can affect user experiences related to immersion, specifically the presence aspect.
The study's findings can be used to understand how different locomotion methods impact the user experience in terms of immersion.
The next experiential construct to be discussed is flow, following the discussion on immersion.
It is recommended to watch the KAI presentation for the paper, which was published at the Human Factors in Computing Systems academic conference, to provide better context for the discussions in the video.
Flow
Flow is a mental state of extreme positive engagement where a person loses self-consciousness and experiences high concentration, yet effortless and spontaneous performance of an activity, often associated with a feeling of happiness.
Flow is often used to explain game experiences, such as playing a favorite massively multiplayer online game (MMOG) or even simplistic games like Flappy Birds, as it can lead to addiction.
The concept of flow was first introduced by Mihaly Csikszentmihalyi, and it has eight dimensions or subconstructs: a sense of clear goals, a level of challenge matching one's skill, complete concentration, loss of self-consciousness, sense of control, effortlessness, transformation of time, and an autotelic experience.
A sense of clear goals involves knowing what to do and receiving immediate feedback, while a level of challenge matching one's skill means the task is not too hard or too easy.
Complete concentration on the task at hand means being oblivious to interactions, and loss of self-consciousness involves losing awareness of oneself and the surroundings.
A sense of control involves feeling total control of actions and decisions, and effortlessness means feeling at ease in performing even difficult actions.
Transformation of time involves feeling like time is warped, and an autotelic experience means being intrinsically motivated by the action itself, not by external factors like money.
Flow can be measured using well-validated questionnaire instruments, such as the Flow State Scale 2 or the Flow Short Scale, which consolidate the eight dimensions into three abstract subconstructs: overall flow, absorption by activity, and fluency of performance.
In a study on a VR commuting simulator, the Flow Short Scale was used to find differences in flow states among participants in different locomotion experimental conditions.
The results showed that overall flow was somewhat the same across all conditions, but participants felt they performed more fluently in the heat bombing condition versus the full body condition.
Cybersickness
Cyber sickness is an undesirable experience that refers to symptoms of sickness due to cyber activities in immersive systems like AR and VR, including nausea, dizziness, disorientation, and other physical discomforts.
The most common way to measure cyber sickness is by using validated questionnaires, with the gold standard measure being the Simulator Sickness Questionnaire (SSQ), which was derived from a motion sickness questionnaire in 1988.
Despite the SSQ being widely used, there is a need for more varied and adequate measures for the varying types of immersive experiences, and newer instruments like the VR Sickness Questionnaire (VR SQ) and the Cyber Sickness Questionnaire (CSQ) are gaining traction.
These instruments have subconstructs that group items into symptom categories, such as oculomotor symptoms and disorientation symptoms.
A study on users playing Valve's original Half-Life 2 game in both desktop and VR conditions found that cyber sickness negatively affected immersion most of the time, but sometimes seemed to enhance immersion.
The study collected both quantitative physiological data and qualitative think-aloud accounts from players, which found that cyber sickness sometimes intensified feelings of immersion and presence.
The relationship between cyber sickness and immersion is not yet fully understood, and there is no clear conclusion on how cyber sickness interacts with immersion.
The general understanding of cyber sickness is also shallow, and there is no clear understanding of the causes and relationships with related factors, such as hardware specifications, immersion factors, and demographic factors.
The most well-accepted theory is that higher visual-vestibular conflict causes more intense symptoms of cyber sickness, referring to the disparity between what is seen visually through the device and what is experienced physically.
The vestibular system, a set of organs found in the inner ear, plays a crucial role in detecting motion, position, and orientation, and can sometimes conflict with visual information, leading to visual-vestibular conflict.
To reduce this conflict, VR application design guidelines recommend matching real-world actions as closely as possible to a person's view of the virtual environment viewed through the device.
A study using a VR commuting simulator for LTA employed a measure of cyber sickness, using the VR SQ, and found that overall cyber sickness goals were very low across all conditions.
However, participants who experienced cyber sickness showed slightly higher sickness in the full-body condition, particularly for oculomotor symptoms related to the eyes.
The study also found that arm swinging had the highest sickness, while leg lift had the lowest, which aligns with the understanding of visual-vestibular conflict.
In the Lake Leaf condition, participants moved their legs more, aligning with the walking action viewed through the HTC Vive HMD, resulting in lower sickness.
In contrast, the Arm Swing condition caused more disparity between the participants' movements and what they saw in the Vive, leading to higher sickness.
Affordances
Affordances is an important HCI concept coined by psychologist James Gibson and made popular in Donald Norman's book "The Design of Everyday Things" that provides a useful conceptual design tool to relate system properties to user experiences.
The concept of affordances refers to the relationship between the properties of an object and the capabilities of an agent, determining how the object could possibly be used.
A famous example to understand affordances is Norman's doors, where the presence of a knob on a door provides a visual signal that affords turning, and the user's experience with how things work in the world contributes to this affordance.
Other examples of affordances include stairs affording climbing and underlined text on a website affording clicking.
Norman's door example also highlights how affordances can sometimes conflict with the intended use of an object, such as a door with a handle that affords pulling but is meant to be pushed.
In VR applications, affordances can cause usability problems if objects that look like real-world objects do not have the expected interactions implemented.
Understanding the concept of affordances has a profound impact on user experiences, and it will be revisited in the context of immersive applications and interaction designs.