Random Musings and a Case Study: Michael Naimark and the Development of Spatial Augmented Reality

Author's note: This article was written as an assignment for DEA2510: History of Design Futures, a class taught by Chad Randl for the College of Human Ecology at Cornell University. This makes it a little more constrained to the deliverables of a prompt as opposed to natural paths of inquiry. It seeks to reflect on an innovation of a predictive nature presented between 1985 and 2005 and trace the development following that prediction to the present, in this case tracing Michael Naimark's 1984 “Spatial Correspondence in Motion Picture Display”. The subject of spatial augmented reality is a hot topic within contemporary themed entertainment, and as such I felt it is worth including on this site. I do not claim for this history to be comprehensive, rather I find that its value is in its focus on the trajectory of Naimark's ideas, specifically.

Spatial augmented reality, more commonly known by its commercial name “projection mapping”, is the practice of integrating synthetic information onto the real environment, generally through film or video projection onto complex surfaces (Bimber and Raskar, 2005). The technology is the product of an intersection of the larger histories of film projection and virtual reality but has split to become a separate technology in its own right (Jones, n.d.). This paper will examine the earliest theorized example of spatial augmented reality, Michael Naimark’s “Spatial Correspondence in Motion Picture Display”, as well as the trajectory of the technology’s development compared to Naimark’s predictions for its future.

The first instance of spatial augmented reality onto a surface more complex than simply flat or spherical was in 1969 with the opening of the Haunted Mansion attraction in Disneyland, California. Sixteen-millimeter film was projected onto blank tombstone busts in a graveyard setting, making the busts appear to be living faces singing along to the theme song of the ride (Jones, n.d.). The practice was not truly theorized until 1984, however, with the publication of Naimark’s “Spatial Correspondence in Motion Picture Display” in the SPIE Conference Proceedings. Based on his experiences in the production of the film installation piece “Displacements”, Naimark details how spatial augmented reality could be used to make “moving movies”, where movie projectors move around a space while projecting in a similar manner as movie cameras do while filming. In “Displacements”, Naimark created a moving projector system on a set of an archetypal American home that had been spray-painted white to act as a relief screen (Naimark, n.d). The result is a three-dimensional film experience that integrates the nature of the projection space into the film itself (see Fig. 1).

Figure 1. Michael Naimark’s “Displacements”, in which a narrative is projected onto a 
three-dimensional set via a moving projector (Today and Tomorrow, 2008).

Naimark describes in the paper his prediction for the future of the film experience, where “the ultimate media room will be indistinguishable from ‘reality’”. He describes how a “video flashlight” would bring a level of interactivity to the moving projector, where the user “decides what to see” by pointing the device to see the film projected onto the environment before him. The device, essentially a handheld projector, would relay movement and directional information to a computer housing a three-dimensional film world calibrated with the environment around the user. Essentially, the device would understand what the viewer is looking at and project the corresponding part of the film world onto that space. “The viewer IS the camera operator,” Naimark writes, “it is an interactive system”. For Naimark, spatial augmented reality is the key towards film progressing past a passive two-dimensional experience into an interactive three-dimensional one.

Following Naimark’s predictions, the trajectory of projection mapping split into two technologically different paths: fixed-position projectors and mobile projectors. The design questions inherent in the mobility problem make these two paths fundamentally separate, with the former playing a role in entertainment settings and the latter finding uses in varied industrial and professional fields. Though the development of the latter tends to align closer with Naimark’s “video flashlight” system, the former is more relevant to the evaluation of his predictions given their frequency of use in entertainment, Naimark’s field of interest. 

The first historical node that would define the use of fixed-projector spatial augmented reality was the patenting of “Apparatus and method for projection upon a three-dimensional object” by Marshall Monroe and Willian Redmann for the Walt Disney Company (U.S. Patent No. 5,325,473, 1994). What differentiated Monroe and Redmann’s method from Naimark’s was the inclusion of a user-controlled sensor system that generated input data affecting the projection display. The patent describes a scenario where the user uses a special paintbrush that, when pointed at the complex shape being projected upon, could signal the projector to “paint” the shape on cue with the user’s movements (see Fig. 2). Rather than Naimark’s movable “video flashlight” illuminating pre-recorded footage, Monroe and Redmann’s system uses the projection space itself as the site of interactivity.

Despite the novelty of interactivity in Monroe and Redmann’s system, there was still the central problem of how to accurately illuminate a three-dimensional object. Where the Disney patent was able to correct problems of distortion produced with two-dimensional projection on three-dimensional objects, it was unable to do so from beyond a single known user perspective. The answer to this problem would come in 2001 with Ramesh Raskar’s “Shader Lamps”. Working with a team at UNC Chapel Hill, Raskar produced a model in which several projectors could be arranged around a highly complex object as to render its surface completely, treating illumination “basically as a 3D perspective projection from a lamp” (see Fig. 3). This innovation allowed for pronounced realism and new scales through which to use the technology, its users no longer relegated to a single point of perspective (Raskar et.al, 2001).

Figure 2. Monroe and Redmann’s interactive projection system for Disney. Note the user’s 
“paintbrush” pointed at the object of projection (U.S. Patent No. 5,325,473, 1994).

Figure 3. Raskar’s “Shader Lamps”. The top left image is a model with normal illumination,
 while the bottom right is with 3D perspective illumination (Raskar et.al, 2001).

It would be through the work of Monroe and Redmann and Raskar that the contemporary form of fixed-projector spatial augmented reality would reach its current sophistication. Though various researchers have improved the technologies in the time since their publishing, the elements of Monroe and Redmann’s user-directed interactivity and Raskar’s three-dimensional illumination compose the basic building blocks of contemporary use. Because of the high installment costs of fixed-projectors, unprecedented scale of projection via Raskar’s “Shader Lamp” technique, and the power of the Disney Company in setting precedent in the entertainment industries, the most common use of fixed-projector spatial augmented reality is in entertainment. Examples include the “Seven Dwarfs Mine Train” at Disney’s Magic Kingdom, in which robotic characters have facial expressions projected upon them in coordination with their movements; Box, a film by Bot & Dolly that uses projection mapping in conjunction with robotic arms to create visual effects without post-production; and Lighting the Sails, a video projection project by Urban Screen onto the Sydney Opera House’s iconic “shells” (see Fig. 4, 5, and 6, respectively).

Figure 4. The “Seven Dwarfs Mine Train” at Disney’s Magic Kingdom in Florida. The face of the animatronic 
pictured is completely projection-rendered via spatial augmented reality (Stroshane, n.d.)

Figure 5. Screenshot of Box, a short film by Bot & Dolly. The brightly textured rectangles in the 
foreground are rendered via spatial augmented reality (The Creators Project Team, 2013).

Figure 6. Lighting the Sails by Urban Screen, a projection mapping project on the Sydney Opera House. 
The show is animated to make the building’s iconic “shells” change in color and texture (Urban Screen, 2012).

The development of mobile spatial augmented reality has yielded significantly different uses. Raskar’s early work at UNC Chapel Hill on “The Office of the Future” (1998) used fixed but pivoting projectors to project a workspace computer interface onto irregular surfaces (see Fig. 7). The caveat to their work, that correct perspective relies on the user being in a fixed, known position, would push Raskar to explore mobile projectors as an alternative. In 2003 he developed the “iLamp” with a team at the Mitsubishi Electric Research Lab, a portable, environment-aware projector that individually is shape-adaptive and object-recognizing and collectively can interact with other projectors to form a single image without manual calibration (Raskar et.al, 2003). In 2004, he presented a refined version for specific industrial uses, working again with a Mitsubishi Electric team to develop “RFIG Lamps” (Raskar et.al, 2004). These handheld projectors can sense wireless tags in their environment and process data related to those tags in their visual output (see Fig. 8). These mobile technologies have significantly lower instillation costs than their fixed counterparts due to differences in the scale of projections, and as such they often tend towards industrial and professional uses.

Figure 7. Raskar’s “Office of the Future”, in which multiple overhead projectors transform a 
irregular surface into a workspace. Note that the viewer must sit in an exact location to prevent distortion (Raskar et.al, 1998).

Figure 8. Raskar’s “RFIG Lamps” in a warehouse scenario, where wireless tags within boxes relay information to the handheld projectors,
 allowing them to visually describe the otherwise-unmarked boxes by coded characteristics (Raskar et.al, 2004).

What is particularly interesting about this dual chronology of development is that both incorporate elements of Naimark’s original predictions. His claims of a new interactive and immersive film experience are best seen in the development of fixed-projection spatial augmented reality, which allows for a scale more attractive for themed entertainment and cultural events. His ideas of a “video flashlight” have not appeared in this fixed-projector chronology but rather in that of mobile projector spatial augmented reality, where examples like Raskar’s “Office of the Future” and “RFIG Lamps” demonstrate how the economy of mobile projectors have potential within more corporate and industrial environments. Thus, though both his “immersive entertainment” focus and “video flashlight” system can be seen in contemporary uses of spatial augmented reality, the distinctions between fixed and mobile projectors and their diverging narratives of development qualify Naimark’s predictions significantly. Nevertheless, the true value of Naimark’s work is in his astute awareness of the avenues in which the technology could progress, and the influence of his insights is visible in works of spatial augmented reality today.

            _____________________________________________________________________________

Works Cited

Bimber, O. and Raskar, R. (2005). Spatial Augmented Reality: Merging Real and Virtual Worlds. Wellesley, Massachusetts: A K Peters.

The Creators Project Team (2013, September 24). “[Video Premiere] Projection Mapping and Robots Combine in Bot & Dolly’s New Film.” Vice.com. Retrieved from http://thecreatorsproject.vice.com/blog/video-exclusive-bot--dollys-the-box-unpacks-a-radically-new-design-concept

Jones, B. (n.d.). “The Illustrated History of Projection Mapping.” Projection Mapping Central. Retrieved from http://projection-mapping.org/the-history-of-projection-mapping/

Monroe, M. and Redmann, W. (1994). U.S. Patent No. 5,325,473. Washington, DC: U.S. Patent and Trademark Office. Retrieved from http://www.google.com/patents/US5325473

Naimark, M. (n.d.). “Displacements 1980-84/2005.” Retrieved from http://www.naimark.net/projects/displacements.html

Naimark, M. (1984). “Spatial Correspondence in Motion Picture Display.” SPIE Proceedings, 462. Retrieved from http://www.naimark.net/writing/spatcorr.pdf

Raskar, R., Welch, G., Cutts, M., Lake, A., Stesin, L., and Fuchs, H. (1998). “The Office of the Future: A Unified Approach to Image-Based Modeling and Spatially Immersive Displays.” COMPUTER GRAPHICS Proceedings. Retrieved from http://web.media.mit.edu/~raskar/UNC/Office/future_office.pdf

Raskar, R., Welch, G., Low, K., and Bandyopadhyay, D. (2001). “Shader Lamps: Animating Real Objects with Image-Based Illumination.” 12^th Eurographics Workshop on Rendering. Retrieved from http://web.media.mit.edu/~raskar/Shaderlamps/ shaderlampsRaskar01.pdf

Raskar, R., Beardsley, P., Wilwacher, T., Rao, S., and Forlines, C. (2003). “iLamps: Geometrically Aware and Self-Configuring Projectors.” ACM SIGGRAPH 2003 Conference Proceedings. Retrieved from http://web.media.mit.edu/%7Eraskar/ Sig03/raskarILampsSiggraph03.pdf

Raskar, R., Beardsley, P., van Baar, J., Wang, Y., Dietz, P., Lee, J., Leigh, D., and Wilwacher, T. (2004). “RFIG Lamps: Interacting with a Self-Decribing World via Photosensing Wireless Tags and Projectors.” SIGGRAPH 2004 Conference Proceedings. Retrieved from http://web.media.mit.edu/%7Eraskar/Sig04/raskarRfigLampsSiggraph04.pdf

Stroshane, M. (n.d.). [Photograph of animatronic projection in Seven Dwarfs Mine Train]. Retrieved from http://www.wftv.com/gallery/entertainment/seven-dwarfs-mine-train-magic-kingdom/gCKZy/#5190372

Today and Tomorrow (2008, May 28). “Displacements”. Retrieved from http://www.todayandtomorrow.net/2008/05/28/displacements/

Urban Screen (2012). “Sydney Opera: Lighting the Sails”. Retrieved from http://www.urbanscreen.com/lightning-the-sails/