Confirmation Panel 2022

Introduction to the Project

Through my PhD Research, I am investigating Alternative Designs for Perception and Interaction in Virtual Spaces.

Digital space has been dominated by systems designed for sight based interaction as a primary sense throughout its history. Perception is not only about the senses we access information through, but also how these senses enable us to interpret the experience, and understand it. Interaction in Virtual space is possible through so many different forms, and by diversifying these further, we only increase accessibility for wider audiences.

Regarding and designing digital things as Spaces, rather than flat layers on our screens is important to create immersion within the virtual and in doing so, we increase people’s ability to engage. Alternative Designs are especially significant as corporations begin to see the value of spatial web environments which aren’t game based, and enabling them now allows for varied sensory access from the outset.

This varied access enables people who cannot rely on the visual as their primary sense to enjoy and understand virtual spaces in the same way they can the physical, through a variety of other senses. A Research through design process is being undertaken, leading to a gradual change in the direction of the investigation through evolving practice. This research isn’t just aimed at non-visual approaches, it also seeks to include sight when important to the design of digital space, and not just implement it due to expectation as well as the potential to regard senses as a wider range of access points.

Work so Far

This page is on my Blog/Website, but below is a link to my monthly posts detailing my research as well as a breakdown of my research through design process.

Through experimenting with animation at the outset of my research journey, it allowed me to investigate the notions of space playfully with visual outputs. Animation, which I mainly explored through blender is a spatial output when done in a 3D workspace using digital models, scans and key framing to create motion. This exploration with visual delight lead me to want to look deeper into photogrammetry and 3D scanning, along with idea of positioning real world objects within digital spaces to enable them to be manipulated and decrease the borders between virtual and physical environments.

Machine Learnt Landscapes was a project which spouted from this desire to look at senses of materiality and vanity within virtual realms. It began with my undergrad final year work which I then progressed by combining it with my animation based explorations during the earliest stages of my PhD.

Taking images of myself from my phone’s storage, I fed these into a machine learning GAN, creating machine learnt images of myself which could be produced infinitely. From a run of this Machine Learnt GAN, I generated an output of 1000 non-selective images. I used these images to effect deformation maps and texture on 3D objects in animation to create virtual landscapes which may attempt to ‘appeal’ to my own self image.

I then converted the ideas of these animations into a game engine environment. Embodied through a rigged scan of myself shown below. This was submitted to a UCL Media Anthropology Lab Conference. It was accepted for this conference where I talked about the ideology behind a landscape made of myself. I also ran a workshop for their masters students on the basics of modelling, texturing, and shading in blender. A link to the online live space is available bellow.

Several VR Experiments came from this exploration of Machine learning in game space. I wanted to play with senses within an immersive games environment whilst learning some basics of using games engines. Trying Unreal Engine 4, I began to learn node editing enabling me to create a physics bending game based on weight and gravity with a vast array of cubes laid out on a table. Exploring VR made me think about the lack of tactility and the heavy focus on visuals, not only in games, but within the wider digital space we have access to as well as upcoming social Meta spaces that are being advertised.

Using Gather (a web based conferencing system) for Exploration only furthered this thought. Looking at video conferencing in a new way, Gather brought many game elements into the conferencing space attempting to engage users with interactive and diverse spaces. However as I experienced and came to grips not only with using Gather, but making spaces to work within it, I found a distinct disconnect between the spaces we made, and the video conferencing elements overlaid on top of it. This lead me to develop a paper for DRS Bilbao with Paul Coulton, Dave Green and Joseph Lindley (currently pending acceptance) that highlights and discusses the shortcomings and benefits of the Gather platform. The main takeaway from this was again the overemphasis on the visual, leading me towards an interest in sound as a sensory experience on its own as a way to design something in opposition to this shortcoming of digital space.

With sound in mind, I began again exploring in animated space, this time with an audio focus. I started making mockup levels to explore binaural sound in blender using the software skills from my animation experimentation. This aimed to spatialise the auditory with a digital output in mind. After hearing the preliminary outputs of this, I realised there was potential for maze based games which used sound for navigation. I took the experience I had gained through making the Machine Learnt Landscapes project to begin making a game in Unreal Engine. this went from UE4 to an alpha version of UE5 set to release this year due to a variety of experimental sound features made available through it. This game is currently playable and is in early testing stages.

The stage I am currently at is creating this game for an audio only output. Below is this game at a very recent state with the visual element unhidden to clarify its workings. As the user, you play through a series of maze levels, listening for the endpoint sounds. As the levels progress, they become more sprawling, with obstacles that reset you to the start, and walls that direct your movement.

This game is an attempt to test appropriate interfaces for interacting with audio experiences, alongside optimal sounds and sensory bandwidth limits for these systems. While being presented through a game medium, the research aims to be useful for sound navigation in wider digital space, exploring how sound can be implemented with a mechanism first approach rather than falling into sound based tropes of story driven narratives.

The game was submitted as an Extended Abstract to DiGRAs 2022 Guadalajara and accepted for this, alongside a similar PhD consortium submission on the same topic. I have also applied for ethics approval for testing at the university to gain some preliminary user feedback which should take place within the next month. After this, we have plans to organise testing with blind and visually impaired participants working with charities focused in these areas to gain further feedback on the development for a Co-Designed Research through Design Approach.

Conference Submissions

Going forward, the focus of the project may evolve due to a Research through Design, but currently, the plan is to continue developing the game on the following timeline:

February 2022Continue development of game for initial user testing plus paper revisions.
March 2022Initial University based testing followed by further development.
April 2022Further development followed by user testing at MozFest Manchester.
May 2022Writing up findings and transcripts of user testing.
June 2022Workshops with blind and visually impaired users through charities.
July 2022Writing up workshop outcomes and transcripts. Attending DiGRAS talking about the game.
August 2022Revising game further, working out bugs and adapting based on user feedback.
September 2022Paper/s on the games successes and shortcomings using data and user feedback.
October 2022Further revisions based on outcomes of paper.
November 2022Potential release of game more publicly with a consideration for the entire process.
December 2022Consider what the paper/transcripts/workshops uncover. conceive new research through Design.
January 2023Begin compiling thesis with research methodology, reference collation.
February 2023Write up progression of the PhD so far to clarify findings further.
March 2023Consider where last 14 months should go from here in terms of research through design exploration.

Around November 2022 I will very likely reconsider this timeline and decide on the future pathways based on where the exploration of Research through Design takes me up to that point.