The Augmented Reality Gotham Guide

I launched Gotham Guide at the ITP Spring Show in 2009.

The idea was that users could navigate to a mobile webpage that had a map of Manhattan dotted with little QR code markers. When the user then arrived at the real world location, there would be a similarly branded Gotham Guide QR code there. Once scanned, the QR code linked to a video I recorded at that spot with Luke Miller, the founder of Real New York Tours. His company was (and still is) the number one tour in New York according to TripAdvisor.

Here’s an example of the type of video one might find. This would appear on the corner of Grove and Barrow Streets in Greenwich Village:

There was some excitement around the original idea at the time and I was pleased with the overall look and usability of the site. People responded to the visual markers because it reminded them that the places they passed every day might hold more interest than they first thought.

Although my initial objective was to leverage the burgeoning QR code technology, I found that busy New Yorkers responded to the randomness of spying a QR code on a path they walked every day and learning something new about their city.

But there was also one big problem: I couldn’t get the QR codes to stay up very long! Some of the locations were private property, others were in parks and others were just cleaned with vigilance by the New York Department of Sanitation. I tried affixing the QR codes with stickers, putty, bolts and paint stencils. No matter the material, the QR code would either be taken down or rendered illegible for scanning in less than a week.

Today though, I am proud to announce the semi-rebirth of Gotham Guide: (Now) New York’s only Augmented Reality Tour of New York That Started as The First QR Code Guided Mobile Tour of Manhattan. :-)

Augmented Reality (AR) solves so many of the problems I had during the first launch of Gotham Guide. Using an application development platform called Junaio, I was able to reconstruct much of the site’s functionality in very little time. In fact, putting Gotham Guide on Junaio also offered several other advantages that makes it a much more robust and useful tour than ever before:

  • Location Based Tagging: The tour sites are all geo-tagged, so you can find sites are near you based on your phone’s GPS
  • Routes: Once the marker appears on your screen, you have the option of pulling up directions to the location using Google Maps
  • Web and Video links: In addition to the location, each tag can be supplemented with web content and video
  • Speed: Rather than looking for a QR code, scanning it and waiting for the site to resolve, Junaio features a built-in video player and web browser for faster access and streaming

In addition, some of the newer videos feature another Real New York Tours guide, Nathan Kaufman, who was kind enough to help me add a few new locations for the experience. Also thanks to ITP resident Craig Kapp for his help with coding in Junaio.

HOW IT WORKS

  • First download the Junaio application to your smartphone, either through the iPhone or Android app store.
  • Open up the app and search for ‘Gotham Guide’
  • Next, you’ll see a live camera view with a list of locations
  • Once you pick a location, you can pull up the route or just watch the video!

You can find the Gotham Guide app on Junaio by scanning the below QR code with your iPhone or Android device:

Please note that you will be streaming video and using your GPS during this tour, so any associated charges from your mobile service provider may apply.

CONCLUSION

AR offers a much more robust user experience than my original site. But in moving from QR codes to a location based application, there is some loss of the serendipity that might arise from stumbling upon a QR code by accident. It now requires people to actively use the app and search for Gotham Guide. We’ll see how that changes the way people use the content.

Also, keep an eye out for the ITP Easter Egg!

Surveillance Cameras and Art

A BRIEF HISTORY OF SURVEILLANCE

Before the pervasiveness of video cameras, it was difficult for the authorities to follow an individual walking around in a city like New York. To do so required multiple individuals, some on foot and some in cars, each armed with a communications device.

This NPR story explains how surveillance teams used to operate. The detail team would take turns following the target. First, they’d have to find something the target was holding or wearing to identify him. Next they would take turns following the target through the city to avoid being detected or arouse suspicion. Because they were never sure where the target might go, team members had to carry with them a variety of outfits and disguises to better blend in to whatever environment they found themselves.

These techniques are probably still employed in some situations, but for the most part, the grunt work has been replaced with long lens cameras, CCTVs and other surveillance technology. We all understand to some degree that there are cameras all around us. But the mere presence of cameras doesn’t indicate sophistication. In fact the majority of these cameras are stupid, meaning they either are placed prominently for show but aren’t recording, the resolution of the recording is too low to be useful or they lack the ability to pan, title and/or zoom. The thinking is that the mere presence of the cameras is adequate to induce a certain behavior, which is a 200 year-old idea proposed by Jeremy Bentham called the panopticon. That’s the point, though.

The more I read about identity, the less concerned I am with any one aspect of being tracked. At this point, surveillance video mostly represents the body. It means that my physical presence has been detected and recorded in a certain location. For a larger understanding of the implications of my presence in a given place, one would have to cross reference this information with my voice, credit card purchases, loyalty cards, social checkins and other data that I “give off” throughout the day.

I can’t get my web calendar to sync to my phone and computer without downloading one or two additional “bridge” programs, so the idea that one organization synthesizing any of this data in time to affect my experience (i.e. marketing/advertising) doesn’t yet concern me. But I won’t get too comfortable, because the landscape is changing fast. This is the type of technology that I bet is between five and ten years out:

A team composed of Carnegie Mellon University Robotics Institute and the Sarnoff Corporation were chosen to lead the technical efforts by developing an end-to-end testbed system demonstrating a wide range of advanced surveillance techniques: real-time moving object detection and tracking from stationary and moving camera platforms, recognition of generic object classes (e.g. human, sedan, truck) and specific object types (e.g. campus police car, FedEx van), object pose estimation with respect to a geospatial site model, active camera control and multi-camera cooperative tracking, human gait analysis, recognition of simple multi-agent activities, real-time data dissemination, data logging and dynamic scene visualization.

http://www.cs.cmu.edu/~vsam/

Whoa! Nothing makes me want to add three locks to my door and only accept deliveries from Fresh Direct more than this type of research funding.

SURVEILLANCE CAMERAS AND ART

There’s no shortage of artists claiming to work with surveillance footage. The artist Rafael Lozano-Hemmer has some interesting projects. Among my favorites are his “Blow Up” and “Close-up” pieces, which are part of his Shadow Box series. Although Lozano-Hemmer uses the word surveillance to describe the method by which he appropriates video, I’m not sure I would agree. I don’t think people approaching a video in an art gallery have much of an expectation of privacy. Knowing there was a camera embedded in the shadowbox in front of them would not alter their behavior. Like with my example above, it merely indicates their presence.

Many artists like to talk about how they are turning surveillance on its head or turning it back on those that are surveilling, but I’ve yet to see any projects that actually do that. I don’t think a hidden camera is the same as surveillance. Furthermore, to turn it on its head would be to publicly show the people that are monitoring the screens displaying the surveillance camera feeds, and I haven’t seen that yet.

A more fun use of surveillance cameras is the Tango Panopticon, an organized event which instructs people to stand together in front of certain surveillance cameras all around the world and then at the same time…tango!

Another project I really like of late is this WiFi light painting video. I liked this surveillance technique for uncovering what’s all around us but what we don’t often think about or see. In some way, the video makes me think that the filmmakers are spying on these signals being sent all around us and the devices that originate them.

Image credit: Will Varner

I used the unique characteristics of the Kinect images to create still frame portraits of people who were NOT in front of their computers. For these shots, I tweaked the colors, shapes and threshold to create some different types of images.

Kinect Portraits

Using the Kinect to paint was a fun exercise to start out with. I also enjoyed playing around with the brightest pixel.

… Continue Reading

For Computational Cameras this week, we are exploring the use of Augmented Reality (AR) markers in Processing. In addition to my own project, here are 10 ideas on how AR markers could be used.

10 Uses for Augmented Reality Markers

For Computational Cameras this week, we are exploring the use of Augmented Reality (AR) markers in Processing. In addition to my own project, here are 10 ideas on how AR markers could be used:

Magic Tricks

AR markers could be great for creating all types of illusions. For example, tie one on each side with a piece of fishing wire. When the user slowly pulls his hands apart, it reveals a red ball (or rabbit!) floating between his outstretched hands.

Make a Charlie & The Chocolate Factory Path

A path is laid out around a room with AR markers on the floor. As the camera passes over them, different types of flowers, miniature people and crazy colors pop out at the viewer. This could be interesting if the AR markers are of varying sizes.

Put a Hole in Your Belly

First attached a web cam like the PS3 Eye to your back and make sure that it’s streaming. Next, print out the AR tag and put it on your stomach at the same level as the camera. Load the video streaming from the camera when the program recognizes the AR tag, and it will look like you have a huge hole in your body.

Mouse trap

I’m not sure how easy it is to recognize AR markers when they are small, but it could be fun to attach a small AR marker to your hand with some glue and make it look like you have a mouse hanging there. You try to shake him off, he just won’t let go.

3D Picture Frame Stop Motion Animation

Place a few AR tags in an electronic picture frame. As the tags automatically change, so do the images. The cool thing is they can be a lot larger scale than the frame. The images could be anything from a growing plant to a boy hitting a baseball (or something way more creative than I’m thinking about right now).

Wear It

Wouldn’t it be fun to make a suit covered in AR tags? No matter which way your turn, your body could explode with bright colors, videos and an ever changing array of images.

Presentation Tool

It would be memorable to have a clipboard full of AR tags, and each one represents a different slide in your presentation. As you go through the deck, you can change your slides with fun motions that no presentation software animation can match.

Make a Mask

Put four AR markers on the sides of a box, and put the box over your head. As your turn, each side represents a different portion of your face.

Video Battle

I would love to take two speeches and be able to try and “VJ” with them by hiding and revealing the AR markers. I’m not sure if you can pause a video, though, or whether it just starts over each time.

Sasquatch

Too funny not to include.

More examples here

Here are 10 ways people are using face-detection software like Processing with OpenCV

10 Face Tracking Examples

Here are 10 ways people are using face-detection software like Processing with OpenCV:

1. Chiddy Bang, Opposite of Adults :: Cool rap video that uses face-tracking to enlarge the artists’ heads.

2. Burning Skull Face :: Pretty self-explanatory

3. Augmented Face :: I watched this one four or five times.

4. Face Pong :: Uses Processing and Max

5. Loudmouth :: Facial recognition software replaces the actor’s face with a large pair of lips

6. Find Yourself on Flickr :: The artist uses facial recognition software to scan Flickr (12,000 photos and counting) to find possible matches for his face.
Project Link

7. Buying drinks :: A new Japanese vending machine uses facial recognition technology to “recommend” drinks based on the customer’s age and gender.

8. Mark Art :: The Hit Counter is one of my favorite new media pieces of recent months. Every time the box recognizes a face, it increments its counter by one. This piece is a response to the old way we used to determine a website’s importance — by views.

9. Facial Recognition and Type Design :: The artist uses Processing to build an application that plays with typography based on facial movements:

10. Face Obfuscation :: The opposite of the above, these students use facial recognition technology to hide their faces behind pixelated masks:

I used raw footage of the Egypt protests and ran it through a Processing sketch using the OpenCV Face detection library to extract as many faces as possible. Next I blew up the faces until they were almost pixelated, then added back the sound.

OpenCV Face Tracking

For this assignment, I took some raw footage of the Egypt protests and ran it through a Processing sketch using the OpenCV face detection library to extract as many faces as possible. Next I blew up the faces until they were almost pixelated, then added back the sound.

Code:

import hypermedia.video.*;
import java.awt.Rectangle;

float last = millis();
float interval = 100;
int startTime;
int noFaces = 0;
int noFacesPrev = 0;
int dx = 0;
int dy = 0;
int dwidth = 400;
int dheight = 400;

OpenCV opencv;

// contrast/brightness values
int contrast_value    = 0;
int brightness_value  = 0;



void setup() {

    size( 800,800 );

    opencv = new OpenCV( this );
    opencv.movie( "egypt.mov", 320, 240 );          // load movie file
    opencv.cascade( OpenCV.CASCADE_FRONTALFACE_ALT );  // load detection description, here-> front face detection : "haarcascade_frontalface_alt.xml"


    // print usage
    println( "Drag mouse on X-axis inside this sketch window to change contrast" );
    println( "Drag mouse on Y-axis inside this sketch window to change brightness" );

}

void stop() {
    opencv.stop();
    super.stop();
}

void draw() {

    // grab a new frame
    opencv.read();
    opencv.contrast( contrast_value );
    opencv.brightness( brightness_value );

    // proceed detection
    Rectangle[] faces = opencv.detect( 1.2, 2, OpenCV.HAAR_DO_CANNY_PRUNING, 40, 40 ); // 40,40 is the smallest size its looking for

 // draw face area(s)
  noFaces = faces.length;

  if (faces.length >=1 && noFaces > noFacesPrev) {
    startTime = millis();
    println(startTime);
    println(noFaces);
    println(noFacesPrev);
    noFacesPrev = noFaces;
  }
  if (faces.length == 0) {  // if there is no face, start over
    println("no face");
    startTime = 0;
    noFacesPrev = 0;
  }

 if (startTime > 0) {
    println("started");
    if(millis() - startTime >interval && (faces.length >=1)) {
      for ( int i = 0; i= 800){
      dy = dy+400;
      dx = 0;
      }
      if (dy >= 800){
      dy = 0;
      dx = 0; 
      }
      }
      startTime=0;
      println ("clock reset");
      noFacesPrev=0;
    }
  }
}

// Changes contrast/brigthness values

void mouseDragged() {
    contrast_value   = (int) map( mouseX, 0, width, -128, 128 );
    brightness_value = (int) map( mouseY, 0, width, -128, 128 );
}

void keyPressed()
{

  if (key == 's') {
    opencv.stop(); 
    super.stop();
  }
}