Interaction Design – Gesture Recognition through Fiducials

This is the latest project that I have been working on this semester. Its a six week project on Interaction Design. We were asked to select any topic that we are interested in. Initially I took up “Interactive Movie” but later I changed it to Gesture Recognition. Apart from exploring the technology aspects, we read a lot of theory related to Interaction and New Media Design in particular. The books that were suggested to us explained the fundamentals of different aspects of design. One of the other reasons for the book study was to inculcate the habit of reading in us – the engineers especially, who have lost touch or never had the habit of reading. Not only we had to study, but we had to present our readings in front of the class to others so that each one shares his/her knowledge about the particular topic. This was one of the most interesting project I did at NID.

Human Gestures are mainly divided into 3 categories

  1. Facial Gestures
  2. Hand Gestures
  3. Full body Gestures
I selected Full body gestures that are done using combination of two or more body parts like Hands, Head, Neck, Legs etc. 

Phase 1:

I started by studying the Air Marshaller’s Gestures that he/she does to guide the air craft. Then I tried to capture these gestures through blob detection. However the main problem with the blob detection was – The blob were fixed on screen and hence the blob detection depended on the height of the person using it. Also any disturbance on the blob would be captured as motion.

I recognized 2 gestures during this phase

  • Wingwalker / Guide
  • Straight Ahead
Wingwalker/guide

Wingwalker/guide

Interaction design with Gesture Recognition - Blob Detection/collision technique

Interaction design with Gesture Recognition - Blob Detection/collision technique

Straight Ahead

Straight Ahead

Interaction design with Gesture Recognition - Blob Detection/collision technique

Interaction design with Gesture Recognition - Blob Detection/collision technique

Phase 2:

Then I started looking at ways to capture gestures or movements independent of the user’s height or position, few ideas that emerged were color tracking, tracking IR(infrared) LEDs and tracking Fiducial Markers.
Color tracking was too much dependent on the light that reflected the color. The settings that would detect the color during daylight would not work during night under artificial lighting.

Fiducial markers seemed more accurate and dependable choice for detecting movements. Hence I started with basic fiducial tracking getting the X,Y co ordinates of the markers on screen. Initially I got the markers printed on 5 faces of a cube.

Interaction design with Gesture Recognition - Trying Fiducial Markers

Interaction design with Gesture Recognition - Trying Fiducial Markers

Later I got the markers printed on a thick board and attached it to an elastic band, so that I could wear it easily.

Interaction design with Gesture Recognition - Wearable Fiducial Markers

Interaction design with Gesture Recognition - Wearable Fiducial Markers

I recognized a few basic gestures like

  • STOP – right and left hand
  • Turn the knob left and right
  • Come Here / Go Away
  • Zoom In / Zoom Out
  • Salute
  • Pray
Interaction design with Gesture Recognition - Stop gesture with right hand

Interaction design with Gesture Recognition - Stop gesture with right hand

Interaction design with Gesture Recognition - Stop gesture with left hand

Interaction design with Gesture Recognition - Stop gesture with left hand

Interaction design with Gesture Recognition - Pray gesture with both hands

Interaction design with Gesture Recognition - Pray gesture with both hands

Interaction design with Gesture Recognition - Salute gesture with right hand

Interaction design with Gesture Recognition - Salute gesture with right hand

Phase 3:

Once the gestures were getting recognized, I was asked to map some application on those gestures. So I made a 3D interactive globe which can be rotated in any direction and also zoomed in and out.

Interaction design with Gesture Recognition - Interactive Globe rotate along X and Y axis

Interaction design with Gesture Recognition - Interactive Globe rotate along X and Y axis

Interaction design with Gesture Recognition - Interactive Globe rotate along X and Y axis

Interaction design with Gesture Recognition - Interactive Globe rotate along X and Y axis

Interaction design with Gesture Recognition - Interactive Globe Zoom In

Interaction design with Gesture Recognition - Interactive Globe Zoom In

Interaction design with Gesture Recognition - Interactive Globe Zoom In

Interaction design with Gesture Recognition - Interactive Globe Zoom In

Interaction design with Gesture Recognition - Interactive Globe Zoom Out

Interaction design with Gesture Recognition - Interactive Globe Zoom Out

Interaction design with Gesture Recognition - My setup in Studio

Interaction design with Gesture Recognition - My setup in Studio

My desk and Fiducial Markers

My desk and Fiducial Markers

Gesture Recognition in action !

Ishara: Gesture Recognition - Syed Fawaz Ahmed

Ishara: Gesture Recognition - Syed Fawaz Ahmed

Fawaz

Advertisements

About syedfawaz2002

I am Syed Fawaz Ahmed. I am currently pursuing post graduation in New Media Design at National Institute of Design, Gandhinagar, Gujarat. Engineer by profession, Artist at heart. I have done my Engineering in Information Science in M.S.Ramaiah Institute of Technology You can reach me at syedfawaz2002@gmail.com
This entry was posted in design. Bookmark the permalink.

2 Responses to Interaction Design – Gesture Recognition through Fiducials

  1. Ziddarth says:

    Excellent work and that too in not a lot of time. Microsoft Kinect’s success has really boosted interest and applications in gesture recognition. What applications did you have in mind for this? Or was this just an exploratory project?

    • Thanks Siddarth !
      Yes, Kinect is much ahead in terms of mapping natural gestures, body contour tracking (even depth). Fiducials on the other hand is a low cost, open source technology. I have an application in mind for this which I ve been asked to work for 3 more weeks(in parallel with the normal classes), lets see what I can do….

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s