Redefining whats real

Augmented Experience

Redefining what’s “real”

This blog post will not follow the format I have been using so far. I am stepping away, for a moment, from the safety of the informational spectrum into the shaky territory of philosophical debate. During the semester we have seen and demonstrated, explored and critiqued the variety of current and coming technologies as well as their creative applications… and some not so creative ones. So allow me summarise and consider the media arts spectrum at a deeper level.

Thinking back I tried to recall my first experience with augmented reality but mostly came up with movies depicting some form of augmented reality. Fair enough, many camera tricks, costume/ makeup gimmicks and anything involving projections could qualify for this. In this case one could go back as far as the various ghost creation techniques used for theatre productions. But there is one movie in particular that stands out; They Live  written and directed by John Carpenter in the late 1980’s. In this movie a wandering “no-hope-er” discovers a pair of glasses. When he wears these glasses they filter out all of the layers of communication and show only the base message for example; obey, stay asleep, marry and reproduce, consume. As the wanderer looks around the city the billboards, magazine covers, shop signs and eventually some of the people on the street fall victim to these truth glasses.

The scenes of this movie depict pulling back the layers of communication whereas, in this post internet society, with its growing array of wearable technologies, we seem to pile these layers on. The virtual experience is fairly easily definable as ‘non-reality’ but the augmented experience is the somewhat dangerous territory of not being able to fully discern the real from the virtual. There are a plethora of Star Trek series and episodes that utilise teleportation, the “holodeck” (full scale, interactive holographic chamber) and the contact with alien worlds/races. There is a moment towards the end of the Star Trek feature film Insurrection (Frakes, 1998) where a portion of the crew are completely unaware they were transported into a holographic projection. It was a moment where reality and the virtual faced off because the line between virtual and real had utterly disappeared. The Matrix (Wachowski brothers, 1998) made this blurred line even more of a potential threat, depicting our ‘real’ as the fully augmented situation. The truth glasses of They Live concur that, really, the world is full of augmentation without the use of or need for extra technologies. People in advertising, branding and ‘news’ media get paid obscene amounts of money* to accomplish this augmentation on a daily basis.

When I was growing up there was a campaign by Nickelodeon children’s television channel (during the mid 90s) that I remember well, I’m going to try and quote as much as I remember- it may not be verbatim;

First there was computer games, then came virtual reality, but now there’s something that blows virtual reality completely away- Actual Reality. Feel the weight of the ball; because you’re actually holding the ball. Feel the wind as you move; because you are actually outside….

I have always had an interest in martial arts, fascinated by Mortal Kombat (Anderson,1995) and especially the chi balls. The movie is hard to watch now due to the bad acting (or bad scripting) but when I actually started martial arts, training in a regular class, my world came to life. I had thus experienced ‘actual reality’. I had come full circle.

In coming full circle from reality, through augmentation to virtual and back again, as experienced through cinema, we can look at the augmentation of reality as a spectrum of critique. To understand that the world is already presented in augmented form is to start asking the question of ‘what or where is reality’ which is beyond the scope of this intermediary piece of writing and will be left to the philosophical experts.

 

*BP oil spent 211 million dollars on their brand update. Most expensive brand ever recorded- Symantec at $1,280,000,000 (Canon, 2012).

 

References

Anderson, P. (1995) Film; Mortal Kombat. Threshold Entertainment. USA

Carpenter, J. (1987) Film; They Live. Alive Films. USA

Cannon, M. (2012) Top 5 most expensive logos of all time. Website http://www.graphicdesign.com/article/top-5-most-expensive-logos/ sourced 16/10/16

Frakes, J. (1998) Film; Star Trek- Insurrection. Paramount Pictures. USA

Wachowski Brothers (1998) Film; The Matrix. Village Roadshow Pictures, USA

Interactivity

Life is an interactive experience (anon.)

In the Disney animation production The Hunchback of Notredam (1996) a character makes the following observation “Life is not a spectator sport”. This is something of a philosophy perhaps but true to my experience. In my study of structure and level design of digital and interactive games I have come to understand the varied and numerous levels of interactivity and all of these levels can be applied to the processes of life. This is expounded below.

This philosophy is backed up scientifically by the work of Deci & Moller ( 2005) with regards to competence motivation. This may at first seem off topic but our engagement and interactions with objects (whether real or fictional) almost always can be traced to that sense of satisfaction achieved through attaining competence at something. If something in interactive then we have the capacity to decide the next step in some way. Human- technology interfacing could be as simple as a switch or button, a mouse or joystick, or as complex as motion tracking or sound triggers.

The levels of engagement and interaction are closely interrelated and seem to range from the point and click adventure, where you do little more than what the industry slang refers to as “quicktime events”* (which could equate to being an automated response to an alarm in the morning which engages a habitualised process called the morning routine), right up to high level engagement called agency of which, surprisingly enough, there is very little available in modern society**. Agency, in essence, with regards to interactivity, is the capacity for and variety of “meaningful choice” (Worch, 2014).

There has been studies into the functions of interactivity as a learning implement by Sweller (1994) which has been used extensively in the games industry (Bjork, 2004). And, more recently, is beginning to be seen in the art world (Alvarez, 1998) and more specifically in media arts (Reiser & Zapp, 2002, Graham & Cook, 2012). In such media interactivity can be used as part of the medium, the fact that the viewer/ user is making the choice and making affect upon what’s before them is part of the comminuique. This signified that interactivity in art is moving beyond mere engagement practices into the more experiential.

 

* “Quicktime event” in game industry slang is a derogatory term used to describe a point of interaction which only has a single outcome- generally initiating a pre-rendered video or action. It comments back to the early game building days of Macromedia Director which exported games as Quicktime executables.

* The observation of there being very little agency in modern life comes from previous studies I’ve made into the symbolism of the nuclear family and prescribed life path of, and this isn’t verbatim, get good grades, get a good job, find a good wife, have good children (based off the words of Kiyosaki, R in his seminar/lecture “Rich Dad Poor Dad”). We are led to believe that the biggest “choice” in life is the career path but the vast majority of us plebs must settle for what we can find, not what we want to do.

 

References

Álvarez, F. (1998). Waterfronts of Art Technologies of Information, Interactivity and Public Art.

 

Bernstein, D. (1997). Creating an Interactive Audio Environment. Gamasutra: http://www. gamasutra. com/features/19971114/bernstein_01. htm.

 

Bjork, S., & Holopainen, J. (2004). Patterns in game design (game development series).

 

 

Brandon, A. (1998). Interactive Music: Merging Quality with Effectiveness.

 

Deci, E. L., & Moller, A. C. (2005). The Concept of Competence: A Starting Place for Understanding Intrinsic Motivation and Self-Determined Extrinsic Motivation.

 

Graham, B., & Cook, S. (2010). Rethinking Curating: Art after New Media-Leonardo Books. The MIT Press.

 

Moreno, R., & Mayer, R. E. (2005). Role of guidance, reflection, and interactivity in an agent-based multimedia game. Journal of educational psychology, 97(1), 117.

 

Rieser, M., & Zapp, A. (Eds.). (2002). New screen media: cinema/art/narrative (p. 151). London: British film institute.

 

Vallino, J. R. (1998). Interactive augmented reality (Doctoral dissertation, University of Rochester).

 

Sweller, J. (1994). Cognitive load theory, learning difficulty, and instructional design. Learning and instruction, 4(4), 295-312.

 

Worch, M (2014) meaningful Choice in Game Level Design. Originally sourced from Gamasutra.com, now available at https://www.youtube.com/watch?v=BEF4GVNzkUw

Soundscapes

“When people talk about augmented reality, they usually think it’s about visuals that are put on top of the camera image,” Breiden Bruecker explains to Gamasutra. “It’s always a visual thing. But not many people think that the same can be done with sound.”

 Alexander, 2011

 

Our ears follow two functions, hearing and listening. Hearing is the input of soundwave data and listening is the filter through which that data is passed to turn the data into information so the brain can interpret it. It is the same with our eyes where we see everything but look at specifics. Between light-waves (seeing) and sound-waves (hearing) we formulate our reality. Touch and Smell refine that reality and taste is a bit too specific to warrant inclusion here.

Hearing is one of the human body’s major senses. Beyond this, as it is so closely interactionary with the brain, hearing/ audio is one of the two main streams of learning (Mayer, 2002). The Cognitive sciences have put a great deal of research into this; referring to it as sensory memory (p17-24 .Brunning, Schraw & Norby (2011) modality (p223) or Cognitive Load Theory (p223, Paas, Renkl & Sweller (2003 ). It is often considered that telling someone as you show/ demonstrate is the most effective learning method (Mayer, 2002, 2008). In this way the media arts spectrum, where sound design and visual design have come together, has the potential to be a critical mass of communication. Thanks to computers, televisions and games the vast majority of our recreational time is spent enveloped in audio-visual fantasies. Thanks to organised sound (or what Attali, in Cox & Warner (2003), defines as music) the larger world-scape can be “consolidated’. Also in Cox & Warner is Russolo’s Manifesto where he explicitly includes non-musical sounds to create a less restrictive musical sound.

Sounds have the capacity to transport us anywhere at any time without leaving our seats (unknown). It’s been my experience that books can also transport us but the process of reading is complex, it is very difficult to meditate while reading through the guide but an audial guided meditation can get you meditating in minutes. Kastbauer (2013) discussed an experience he had while reviewing the audio-work for a game he was producing. He marvelled at its capacity to make what was visual real, to turn the sights into an experience. Turner (2015) supported this at GDC Europe when he discussed how the audio plays a role in how a game “feels”. Diamante (2007) wrote, “Recently, Kondo [(Sound Creator for Nintendo)] has been playing with music actually driving the actual gameplay.” To this end it seems that it would quantify that Audio, and Sound in general, plays a critical part in the effectiveness of Media arts; especially when in conjunction with visual elements.

References

Alexander, L. (2011) Dimensions; Augmented reality purely through sound. http://www.gamasutra.com/view/news/127976/Dimensions_Augments_Reality_Purely_Through_Sound.php 23/8/16)

Bruning, R. H., Schraw, G. J., & Ronning, R. R. (1999). Cognitive psychology and instruction. Prentice-Hall, Inc., One Lake Street, Upper Saddle River, NJ 07458.

Cox, C., & Warner, D. (Eds.). (2004). Audio culture: Readings in modern music. A&C Black.

Kastbauer, D. (2013) Envisioning our interactive audio future. http://www.gamasutra.com/view/feature/195324/envisioning_our_interactive_audio_.php (23/8/16)

Diamante, V. (2007) GDC: Koji Kondo’sInteractive Musical Landscapes. http://www.gamasutra.com/view/news/104002/GDC_Koji_Kondos_Interactive_Musical_Landscapes.php (23/8/16)

Mayer, R. E. (2002). Cognitive theory and the design of multimedia instruction: an example of the two‐way street between cognition and instruction. New directions for teaching and learning, 2002(89), 55-71.

Mayer, R. E (2008). Learning and Instruction (2nd Ed.). New York: Pearson.

Paas, F., Renkl, A., & Sweller, J. (2003). Cognitive load theory and instructional design: Recent developments. Educational psychologist, 38(1), 1-4.

Seitz, A. R., Kim, R., & Shams, L. (2006). Sound facilitates visual learning. Current Biology, 16(14), 1422-1427.

Turner, J. (2015) Video: How great sound design can make your game “feel” better. Presented at the Game Developer’s Conference Europe 2015 sourced http://www.gamasutra.com/view/news/253265/Video_How_great_sound_design_can_make_your_game_feel_better.php (23/8/16)