Affordance-HCI Research
#Machine Learning
#Physiological Sensors
#VR/AR
MIT Fulbright Scholar, Researchers
Guzden Varinlioglu, Han Tu
YH. LI
ZH.TIAN
DigitalFutures
CONTEXT + INTELLIGENCE
My Contribution:
concept development, research, 360 site context analysis, proofreading materials, build apk. file, questionnaire design, testing and interview, GSR set up, control experiment process
Tool:
Oculus Quest 2
Agisoft Metashape
Unity
Arduino GSR
Adobe Premiere
Collaborated with:
Timeline:
June 2023- Sept 2023
A Crowdsourcing Tool for Physiological and AI Data Collection in VR
People naturally perceive and respond to environmental information. When an object or space functions in a way that matches what people expect based on their senses, it supports a clear understanding of the environment. But when the appearance of something doesn’t match its real function, it can confuse or disrupt this perception. This mismatch can also affect emotional responses. Our design approach explores how people instinctively react to such situations, focusing on how it influences their attention, concentration, and decision-making.
How can we empower the context in virtual environments with our understanding of the real world?
Overview
Thesis
How does the conflict between environment and sound affects memory, concentration, judgment level, and emotional change?
Abstract
Individuals possess a strong innate ability to perceive environmental information. This means that when an object or a environment's functionality in line with people's sensory expectations, it results in a consistent perception of their surroundings. Conversely, if the visibility of an object or environment feature contradicts its actual purpose and is consistent with the way people perceive such things in real life, or if it alters their natural understanding of objects and environments. So how that might affect their emotional state? Our design approach focuses on dissecting people's instinctual responses to this situations and examining the depth of their concentration and judgment.
Key Word
affordance theory | spatial experience | sensory information
Methodology
What are Affordances?
An affordance is what a user can do with an object based on the user’s capabilities.
An affordance is, in essence, an action possibility in the relation between user and an object.
VR environment Design
We choose a city environment and a nature environment and it sound sources as our dependent variables, and MIT BAKER ROOM as independent variables.
Hypothesis
Exterior environment variables
Sound variables
Interior room variables
In interior spaces, affordance methodology examines how different sounds impact the perceived uses of that space. Does an open plan office afford focus work and conversations when phones are ringing and keyboards are clacking loudly? Background sounds communicate context and environment type to users before they even enter a space. The buzz of voices may afford a welcoming, lively social environment. The sound of vehicle engines and city noise outside affords a downtown, urban context. Familiar interior sounds can create continuity between the exterior and interior environment.
MIT BAKER ROOM
Experiment Design
Reading sections from Data Cities: How Satellites Are Transforming Architecture And Design by Davina Jackson
Data Cities explains how flows of data will inform our future behaviors in physical, virtual, and hybrid-reality situations, and architecture and cities are being reinvented as not merely static structures, but places that pulse.
Proofreading materials
We add proofreading materials in each VR environments. The purpose is to test how environment and sound affects memory, concentration, judgment level, and emotional change. Although the room variable are the same, but the setting inside the room are different
VR environment Setup with Meta Quest 2 + Unity Testing
Discontinuity between exterior and interior sounds can potentially cause perception dissonance that negatively impacts users. If outdoors sounds peaceful but indoors reverberates loudly, the mismatch might produce user stress and discomfort. Affordance methodology aims to choreograph complementary interior and exterior sounds.
Affordance methodology also considers the important relationship between sound and the interior and exterior environment. Sound is an integral aspect of how people perceive spaces and possibilities for action within those spaces.
GSR-skin sensor + Arduino
In addition to the insights gained from the experiment regarding people's instinctual perception and emotional responses to the visibility of functionality in objects and environments, the inclusion of Galvanic Skin Response (GSR) sensor data in the VR experiment provides valuable quantitative information. The GSR sensor data captures changes in participants' skin conductance, which serves as a physiological indicator of emotional arousal.
Questionnaire
We conducted a research questionnaire that combine a mix of close-ended questions and open-ended questions, mostly about their feelings and experiences in VR.
Get Ready for the experiment
“Thank you for interesting and accepted for our research experiment！”
Step 1: You will need to sign a consent form and fill out the Google form about your basic information(name, age...)
Step 2: Choose one of the VR environment (4)
Step 3: Put on VR devices and GSR senor
Step 4: 2-3mins (exploring and get familiar with the vr environment), 2-3 mins complete the proofreading task
☺Adjust your headset until you see clearly
☺Be relax
☺A screen record will be on for your vr process
Step 5: take off the headset and GSR sensor
Step 6: fill out the Google form about your feeling
Step 7: user interview
you will watch your recording and tell about what you did, what you think and what you feel during the VR process.(not what you saw because I know how the interior and exterior look like, act as if you are alone, speaking to yourself as you solve the task.) you need to keep talking during you are watching
Step 8: get ready for next VR experiment!
If the process goes smoothly, probably will take you 15-20 mins for each VR environment. Thanks for you time!
Collection
&
Analysis
I interviewed 8 people, and they are…
During VR experiment…
Interviewing…
Collecting…
They said…
GSR Data Comparison
When an object’s appearance contradicts its actual function, Galvanic Skin Response (GSR) data often shows increased skin conductance. This suggests stronger emotional reactions, such as confusion or frustration. When appearance and function align, GSR levels are lower, indicating a more stable and calm emotional state.
In our study, different virtual reality (VR) environments showed how sound and visual cues affect memory and attention. In condition C2, a calm setting with pleasant sound led to a gradual decrease in GSR, suggesting reduced emotional arousal. However, attention shifted during a proofreading task, causing a temporary rise in GSR, which later returned to normal.
In contrast, C1 featured a noisy and visually busy environment. Although the sounds were loud, participants felt comfortable because the visuals matched their expectations. This alignment reduced the negative impact on memory despite the noise.
Condition C4, however, presented a mismatch between visuals and sound. The loud noise, combined with unexpected scenes, caused discomfort and disrupted focus. GSR data showed higher arousal levels, indicating stress and reduced memory performance.
These results highlight the importance of consistency between visual and auditory cues in VR. Mismatched sensory input can negatively affect concentration, emotional state, and memory. GSR sensors help reveal how these external factors impact cognitive performance. Understanding these patterns can help us design more effective and emotionally supportive VR experiences.
In this context, VR scenes and sounds act as affordances. A calm environment like C2 supports focus and relaxation. In contrast, environments like C4, where elements clash, create distractions that reduce cognitive effectiveness.
Conclusion
In this study, when the environment and sound are not the same, it generate a heightened sense of curiosity in individuals. This research sheds light on the people's instinctual perception of environmental information. Our experiment recognizes the significant emotional changes that can occur when the visibility of an object or environment's functionality contradicts its actual functionality, thereby disrupting users' visual and auditory cognition or altering their inherent understanding of objects and environments.
Gibson, J. J. (1980). The ecological approach to visual Perception. The Journal of Aesthetics and Art Criticism, 39(2), 203. https://doi.org/10.2307/429816
Preece, J., Rogers, Y., Sharp, H., Benyon, D., Holland, S., & Carey, T. (2003). Human-Computer interaction. In CRC Press eBooks. https://doi.org/10.1201/9780367804787
References
Reflections
Numbers of samples I will continue to interview more people for this experiment.
Model variables: I did 3D scanned another room model, however, the file was too big to bring into the VR devices, we may bring other variables add-on to interior environment.