Rich Sensors

Rich sensors, as a project, looks at using advanced sensors, rich in information to fuel digital outputs visible on the computer. The first example used is using an android tablet with processing on macintosh. These combined allow the accelerometers on the tablet to wirelessly effect the visual output on the desktop.

Screenshot 2018-11-27 at 15.33.23

Once the applications are loaded on both devices, the sketch will run and when the IP is input and the devices are both on the same network, the sketch will function seamlessly between both devices allowing for a rich and variable input.

IMG_3832

This video below demonstrates the sketch running with the device being manipulated in 3D space and the sketch on computer reacting accordingly.

Using touch osc within unity allows the gyroscope within my phone to move an object within a virtual environment within unity. This link to the virtual world, through real world movement feels odd at first, but also somehow very intuitive. I want to experiment with both the real world and virtual outputs, allowing the experience to feel oddly linked.

I feel that unity is the output I want to take due to it allowing me to experience virtual linkage in a whole new way.

 

After creating the virtual cube you see at the end of the video, I realised that I want to create a semi surfaced cube, potentially creating a bump map to add external detailing to the surfaces. This would allow me make something which exudes meaning and almost creates a virtual artifact, similar in the way to how I wanted to create virtual emotion in the design domain project.

These bellow drawings show the kind of shape which I wanted to create for the viewer to be able to manipulate in the virtual environment they are placed within.

Here you see a similar idea with shapes onscreen being moved via the use of a wii remote and balance board, this allows for multiple inputs to be mapped more cheaply, with more complicated output variables then being accessible to the user.

After experimenting with different ways to form an output, I feel that I want to create a form of three-dimensional digital puzzle. Potentially this idea takes several different puzzle forms and amalgamate them into something which may at first appear merely as an abstract shapes rotating in unknown space, but once investigated deeper reveals a complex series of three-dimensional images which the user can form.

Image result for face optical illusion old womenImage result for 9 piece slide puzzleImage result for jigsaw puzzle

This idea somewhat forms from the three above puzzle types and a test of several shapes rotating at different rates is shown bellow.

These are all controlled by one input device, yet only all sync up at one angle on that devices gyroscopic sensing. I feel that taking these old puzzles, and forming them into something which is much more rich visually allows deeper explanation. I want to attempt to make several different forms which the pieces could form into. However this could become difficult so the puzzle may remain an abstract assembly.


Here I have created a physical form of the idea I previously described. Using a similar generation method, made a series fo 8 cubes with paths running along their surfaces. These fits together with a series of none connecting similar paths running along the remaining three sides. Once they are properly meshed together, they form sealed line loops which creates a finished shape.

Here are several videos of me creating the final shape. I feel that the way I create the interactivity from this will define how well the interactive experience works. I want it to have no ui, and to remain a clean, rather tranquil experience. I think this means it should all be accessed via motion control, although I am not sure how many controllers I am going to connect and use to operate it yet.

Here is several images to show the connectivity I am using to fuel the interaction with unity. I have imported the models I made into the unity environment and then placed them in the layout they were in mudbox. I then used osculator to get data from each device, routing each wii remote to 2 diagonal cubes. This allows the four wii remotes shown to control the 8 cubes, with the phones accelerometer being used to orientate the camera, allowing for a smooth visual as the other blocks are positioned in order to try to finish the puzzle.

As you can probably see from this video, the idea is nearing completion. I feel that there are only a few changes which would perfect the concept and allow it to be a fully formed prototype concept which can be easily understood. These are as follows:

  • Make the pieces rotate from the center, rather than the center of the 8 cubes as a whole.
  • Form a less distracting skybox to simply the experience and make the experience more aesthetic.
  • Disillusion the link between each pair of rotating objects making it harder to realise which  cubes you as the player are controlling.
  • Create cube housings for each remote to allow them to be placed within a space and appear less digital. These could be simple white cubes which the user can place wherever they want within the space to orientate the puzzle.

This video shows the less distracting skybox and lighting system which I make for the environment. I used several of the textures for the meshes and merged them together. This creates an interesting but also more subtle skybox for the model to sit within. It also makes the camera generally feel more smooth which is what I wanted.

Screenshot 2018-12-12 at 12.42.02.png

  • Make the pieces rotate from the center, rather than the center of the 8 cubes as a whole.
  • Form a less distracting skybox to simply the experience and make the experience more aesthetic.
  • Disillusion the link between each pair of rotating objects making it harder to realise which  cubes you as the player are controlling.
  • Create cube housings for each remote to allow them to be placed within a space and appear less digital. These could be simple white cubes which the user can place wherever they want within the space to orientate the puzzle.

Here I have shown how I changed the cube forms so they would rotate around their own center. I positioned a cube with mesh render disabled within each of the eight visible puzzle pieces. I then added the accelerometer control to those pieces instead, removing it from the shapes, and then patented the shapes to these to allow them to rotate around the cube, which was centered inside them. This gives the illusion of them rotating around themselves close enough that no user could ever tell if they were a pixel or so off-center. It also rotates more central due to the added tubes on the mesh not effecting the center points of them. Once I did this, I offset each of the cubes rotations so that they don’t ever line up when all the controllers are flat. This means the user has to manually align then all making the disillusioning of the link between each pair unnecessary because the puzzle is already hard enough with these added features.

  • Make the pieces rotate from the center, rather than the center of the 8 cubes as a whole.
  • Form a less distracting skybox to simply the experience and make the experience more aesthetic.
  • Disillusion the link between each pair of rotating objects making it harder to realise which  cubes you as the player are controlling.
  • Create cube housings for each remote to allow them to be placed within a space and appear less digital. These could be simple white cubes which the user can place wherever they want within the space to orientate the puzzle.

This leaves me with just the challenge of creating housings for each shapes. I also feel I could try to make it so that if the shapes are cubes, the correct answer will correlate to the cubes being flat on a surface at one of their orientations so that one user can finish the puzzle without holding all the cubes at precarious angles, although this in itself could be quite amusing. Finally if I accomplish this, I may try to refine the skybox even further as there is certain areas which don’t seem together as well as I would have liked.

Here you can see a final video of the project working along with shots of the boxes and sensors placed inside them tapes in place. The video shows several of my course mates learning how to use the virtual puzzle experience. I feel that overall this project turned out really well as a prototype for an interactive experience.


I feel that if I were to produce this for an exhibition space, I would enlarge the blocks making them overly sized and making the user chose which ones to place where to affect the visual. I would also put the visual on the wall as a large projection, making the user feel overpowered by the extra-large and extra dimensional experience they were attempting to complete. I enjoy the bewilderment and confusion the user’s experience when attempting to solve something which is just that little bit less tangible than what they are used it. Therefore I feel this project served its intended purpose and provides interesting insight into the way a user regards truly three-dimensional experiences and how we can expand our minds to conceive them better.


POST RE-EVALUATION:

Looking back on this project. I reconsidered the reasoning behind both my stylistic and physical developement of the final work. I realised that I did infact overcomplicate the raw concepts of the idea, which lead to the core mechanical decisions appearing blurred and losing their clear meaning. For example, the colours of the lines upon each face seemed to become random and overbearing due to the complex skybox and shiny surfaces. Therefore I removed the shine and the skybox completely pairing the surfaces down to what we see bellow.

Untitled-1.jpg

This grid view of all 6 sides also lead me onto my next substantial developement. The camera, controlled by one of the boxes which seemed interesting at first had room for much development. I decided to have it instead be locked to faces and quickly snap between these angles in order to reduce the overcomplication of the work and change the perspective we see it from. In similar fashion to a rubix cube, the user may try to consider a single face without ever-moving the camera, believing the have finished this face, however once they realised the camera can be turned, they understand they have only begun to delve into the complexity available. This is the case with any dimensional expansion.

Here you can see a version of the model where the shine is removed the camera snaps. You can understand how one angle may seem to be the complete puzzle until the user realises there is another level of depth to the whole system them aim to complete.

This whole realisation of how perspective shifts our view of a concept came from one small moment. I was flying over England on a plane to France, looking down at the cloud forms and admiring this new angle. Although I’ve always experienced and understood on sunny days when clouds cover the sun, the world becomes shaded for a moment, it didn’t fully connect within my mind. Seeing it from this angle, it became so much deeper, clouds always seem so flat in the sky, simply a layer which covers the sun and moves, but when viewed from above, they appear almost flat as a layer just above the ground. You can see areas which are about to become shaded and those which are about to leave the shade. From this angle, it almost feels as though you have a deeper and more oracle like understanding of clouds when before they seemed to tower above and control the very light that reached your eyes.

img_2920img_2917img_2913

I took these photos which aim to demonstrate this understanding, and how I aimed to apply it to my own work. I think if this concept were fulfilled, the user would be able snap between the angles to view and complete the puzzle, and once they finish it, the camera would become a free perspective, meaning the user could fully understand what they had completed. This creates more revelations for the player to uncover and potentially allows them to consider perspective and dimensions in a new way because of their experience.