AR Storytelling with Hand Puppets using ARKit and 3D gesture recognition

Jyoti Bishnoi, Not A Bot Studio (@methodiva)

This talk covers the latest advancements in hand pose and 3D gesture recognition and puts them to good use by building virtual hand puppets for story-time with your kids!

With new parents like myself, Hand puppets are all the rage. And while physical hand puppets are quite wonderful, they are also limited by how many you can have, and what they can do.

Here I present a new technological breakthrough that my 2 year old daughter is absolutely gaga about:

AI-driven virtual Hand puppets experienced through AR on her iPad!

At the heart of it, it's a custom hand pose recognition neural net that recognizes how you're moving your hand. Virtual puppets render over your hand in AR and follow the movements of your hand, responding to custom gestures with special effects (Example: Elmo yawns with zZz when you open your hand wide). And ARKit lets you and your kids (or students) experience all of this together!

In this talk, I will walk through how we implemented this, going over gotchas that we ran into, as well as some super-fun virtual hand puppet demos!

About Jyoti Bishnoi

Jyoti Bishnoi is a new mom who started Not a Bot Studio to make learning games for kids using AR. Her mission is to bridge the gap between "learning" and "games" using technology. She has presented previously at Augmented World Expo, AWE Nite SF, and local meetups. With her team at Not a Bot Studio, she released littleDots, a balloon-popping AR game to help kids learn counting, and their next game aims to teach kids Physics concepts in AR. She is usually found busy and at work at the confluence of AI, Mixed Reality, and Games.