Playing CoH with gestures

I've made this video showing a gesture recognition system I have been working on (no, it's not Kinetic, I'm just using a cheap webcam) and testing with CoH, among other applications. Playing CoH like that was tiring, but sure felt real cool. The video is featured in my channel:

http://www.youtube.com/user/jlbernardes

Or if you don't fall for the shameless plug, here is a direct link to it:

http://www.youtube.com/watch?v=mABRE17jIp0

or a link to the smaller interval that only shows the Cimeroran "arresting" part, sans text or music, but with better quality:

http://www.youtube.com/watch?v=T_cvPPUo61g

The videos get choppy because I'm using a single computer to do a bunch of stuff, including the recording. And I'm fighting blue enemies because I was more worried about testing the gesture stuff than about showing off build or skillz. But I think you get the idea.

I asked Ocho whether it was ok to publish the test video using City of Heroes, and he encouraged me to tell people about it, so here it goes. I wasn't sure whether to post here, in the fan creations or multi-media forum, but figured that, when in doubt, this is the place.

Hope you like the video.

Nice work Gamma!

Clicking on the linked image above will take you off the City of Heroes site. However, the guides will be linked back here.

Pretty neat but hard to watch both your gestures and the toon using powers at once.

That's pretty cool. It'd be kinda difficult for me to jump up, spin around, kick, and do a backflip to get Eagle's Claw to work though.

That's an interesting development you have there.

However, I do have a question. I have a Dark Melee brute that has Shadow Maul and Sands of Mu. I'm sure that other Dark Melees might have Flurry. Basically, these are the same power with some differences. Would Gestures2Go be able to differenciate between the three?

Thanks all for the compliments. A few specific answers:

@Chaos Creator: I noticed that, too. Even I get lost there sometimes, and I'm pretty familiar with every frame of that test video by now... Hopefully in the future I'll manage to make a video where a separate camera films the player moving more clearly (instead of just showing the tiny feedback window) and hopefully that will help some...

@Night-Hawk07: One of my current projects nearing 50 is a Claws/Regen Scrapper, so I actually thought about that. That's one case when I guess it's better to design gestures that are not all that similar to the power animations, but still similar enough that they are not too hard to remember.

@Blacjac84: My Dark/SR Stalker is one example of a character who often uses Shadow Maul and Sands of Mu. To differentiate between them, we'd have to define different gestures for each power, but again, hopefully similar enough to the animation that it wouldn't be too hard to remember. In this case, for instance, we could do something like a right punch followed by a left punch be Shadow Maul, and a left punch followed by a right one be Sands of Mu. That should get the job done and be easy enough to remember, even if it is a little quirky, I guess.

Bah, I gesture at the game all the time.

/hadtobesaid

See my Guides and Tutorials

I know that music... where do I know it from?

Michelle
aka
Samuraiko/Dark_Respite

Wow, that's some pretty impressive development work there, DrGamma! I take my hat of to you (and accidentally fire off my nova! )

You'll definitely give MS a run for their money there. There's already a community building up around "hacking" the Kinect, so might be worth looking for some of their forums and letting them know about your product!

I'd love to have this for the MA power set - it'd be a great way of working out while playing

That is actually quite impressive. Though I must admit that the first thing I thought about was "A long time ago, in a galaxy far, far away" The other thing I instantly thought about was gorilla arms.

Truth be told, though, this IS very impressive. I just love how that thing can recognise such complicated gestures, and it looked REALLY cool in the "programme's eye." One solid white head, one solid white hand, and it almost looks like a puppet theatre. I'm impressed with not just how well it can track the hand and it's positions, but also how well it can recognise complex motions, such as a full circle or a click.

Trust me, I don't impress easily, but this was impressive. More impressive than any Knect ad I've ever seen, I'll tell you that much.

Quote:

Originally Posted by Arcanaville Samuel_Tow is the only poster that makes me want to punch him in the head more often when I'm agreeing with him than when I'm disagreeing with him.

Thanks again for all the answer and compliments. Here go the specific answers:

@Night-Hawk07, Golden Girl: Gestures2Go only recognizes hand gestures, so it's not the best fit for MA (they didn't make alternate punch animations for -every- power, did they?), at least not as the system stands right now. Detecting full body gestures is a possible and interesting direction for future work... But then again, Kinetic seems to do that quite well already.

@Memphis_Bill: So do I! But most of the time it just ignores me!

@Dark_Respite: I did find out about Zero Project watching one of your videos. I don't remember though, did you use that exact same piece? If you did, it's a funny coincidence, cause I listened to everything they have on their homepage and just thought that piece was the one that worked best for the movie.

@Night-Hawk07 (again): The timing thing gets tricky. It's possible with Gestures2Go, using time as one of those continuous parameters, but it would require a bit of extra coding work on the person designing the gesture interface, compared to recognizing gestures more instantaneously. But the biggest problem is for the user: we've noticed having to wait a while makes gestures a lot harder to perform right, and it makes users feel like the interaction is slower or something.

@FloatingFatMan: What we're doing here is pretty different from what Kinetic does (although I'm excited to get my hand on their hardware to see what I can do with that nifty camera), but I'm publishing a paper or two about it, so I'm sure the MS people are going to see it.

@Samuel_Tow: Yeah, the "galaxy far, far away" part was no coincidence. (Although, honestly, it was one of the only ways I found to add longer texts to the movie using Windows Movie Maker, an old version too, 5.1. And the only fun way. I'm a total noob in this movie editing thing.)

And the gorilla arms thing is a concern. At least you're not touching anything, so you won't break anything like it happened to those poor touch screens. But having tested playing like this for longer periods (like 15 or 20 minute missions... or longer sequences of smaller tests like the one in the movie) I'm quite aware of how tiring it can get. Sure would not work for longer play sessions, but 30 minutes of playing like this every day and I'd probably be thinner. Or get shoulder injuries.

The ergonomics of using gestures like that is one of the things we want to study (and one of the reasons we haven't released the system yet - it's probably coming out as free and open-source once it's really ready).

That was very impressive.

"You got to dig it to dig it, you dig?"
Thelonious Monk

Very interesting, however the lag between the gesture and the actual power execution is quite distracting. I have that problem with all "gesture interfaces", making a gesture will always take longer than just hitting a key.

@Samuel_Tow: In my computer and using a single core, the entire recognition process is taking in average just above 14 milliseconds (and 13 of those are spent on image processing). So it's quite fast for typical hand speeds. The webcam I'm using is really cheap though, with a maximum capture rate of 15fps (and less if the environment is dark and I need more exposure), so fast movements cause some annoying blur. I don't even try to detect hand postures during the movement, just before it starts and after it ends. The again, if I were using a really fast camera, I might not leave enough processing time for other stuff (such as running the game, or recording the video).

@Dark_Respite: It's awesome (and I downloaded it all, too). I particularly like the orchestral and piano stuff but for this video I thought something more rock-and-roll worked better.

@Leandro: Actually, paraphrasing the matrix, "There is no lag"...

Kidding, but what I mean is, if you look closely at the video, it is working exactly as if I were using the keyboard: one power gets queued while the other is still animating. But since the gestures the user makes are similar to the animations, we tend to perceive a lag that isn't actually part of the recognition.

It's funny that we don't really notice the lag if either the input (such as keystrokes) or the response (like when I use a gesture to select enemies in the beginning) are fast, but if both the input and the response take time (like a gesture and a power animation) AND are similar, there is that perception of lag that is indeed distracting. It's quite an interesting result that we had not noticed before.

I'm not sure if that would happen to other users, since no one but me has tested the system more than once or twice but, for me at least, that perception of lag went away as I got more used to the interface.

OK, let me modify the question a little bit, then:

Can your programme track gestures that see the hands interlocked together? What I'm referring to is, speed notwithstanding, something along these lines. One-hand gestures are interesting, but they are limited in scope. Interesting in terms of basic control, such as moving a cursor and producing clicks, but interlocking-hand gestures have the potential of producing an honest-to-god game all on their own.

Quote:

Originally Posted by Arcanaville Samuel_Tow is the only poster that makes me want to punch him in the head more often when I'm agreeing with him than when I'm disagreeing with him.

Ah, no, not really. I still count on the hands being mostly separate for recognition. I -could- try to fool the system and define hand poses that are actually defined by two hands together instead of only one, since I only work with image properties instead of some anatomic model for the hand, but I'm not sure how well that would work. It's actually something I might check, now that you brought it to may attention.

There is an issue I have with the recent fascination that has been booming through out the media about motion detection in gaming. It's really cool and everything...but personally, I don't get the incentive behind it.

Most of us either go to college, or work during the day. We come home, tired and exhausted, and we wanna play a video game.

Normally, this would involve sitting on a couch, or on a computer chair, picking up a controller or your mouse/keyboard and just pressing buttons at your fingertips to get entertainment value from what you see on the screen and what you hear from the speakers.

With motion detection, that activity turns into a physical exercise. And arguably, it's not really "healthy" physical exercise. So it turns into more physical exhaustion and strain, and eventually make the play sessions shorter and more physically exhausting.

I think the OP's post is really cool. And I really do respect them for taking time in making the video and putting thought into this. But this is just something I had to toss forward because it's always been at the back of my head everytime someone mentions Wii, Microsoft Kinect, or PS Move.

So my question is, what would be the incentive behind such a technology other than the obvious cool factor? And this is a genuine question. I'm not being sarcastic/rhetorical.

I actually agree that using gestures is not always the best way to play a lot of games, especially the kind of games I (and I guess most of us) like to play.

Let's start with reasons for the fascination with gestures in general, though, instead of just in games. I'll reach games eventually. (Warning: huge post ahead, no TLR down there).

Interaction people have been fascinated by using gestures, touch and voice pretty much since interaction research and development began, way back in the sixties. That's because using your hands is something -very complicated- that you know -very well- how to do. I mean, you can perform a huge range of complex motions with your hands without even thinking about it, because you've been training your brain to do that since you're born (or earlier, I don't know). It's why babies are so grabby. By the time you actually remember being a real young kid, you already learned most of what you need to know about manipulating stuff with your hands and depth perception with your eyes, so well that you do these very complicated things without thinking about it. So it's supposed to be much much much easier to interact with something in a computer system just like you interact with the real world - using your hands - than using something weird and unfamiliar such as a mouse or keyboard (not to mention one of damned gamepads with tons of buttons everywhere!). Even when you think you're completely familiar with mouse, keyboard or gamepad, you're still much more familiar with your hands. Voice and speech take a while longer to learn, but people are, in general, still better at talking than they are at typing or mouse pointing and clicking. Gestures may come with ergonomy problems, and voice with privacy problems, but those problems don't stop people from wanting to interact with computers just like they interact with the world, to the point where you start to notice computers less and less.

There is also the fact that traditional interface devices such as keyboard, mouse and gamepads are most adequate for 2D interaction. Typing a text = good. Manipulating a 2D image = good. Running around in two dimensions = good. But add the third dimension and these devices become REAL akward and you need some very complicated rules in the interface to deal with it. Grab some flight or submarine combat simulator and get in a dogfight to see how weird it is. Or grab Blender, draw a cube and try rotating it JUST like you want, in three axes, just using mouse and keyboard. Or try the old CoH PvP with unsupressed running and jumping around, I guess (you can still do that in the arena, right? I never did much PvP). It's tough and usually requires a decent amount of training. Now compare that with how easy it is to rotate any object any way you like just using your hands, or to mimic a very complicated and crazy flight path with your hand, and you'll see what I mean about hands being much better for REAL 3D actions (most of the time, though, you SEE things in 3D, but are bound to a deformed 2D plane - the floor, so you don't really need 3D interaction).

Finally, there are a lot of situations where having a keyboard, mouse or gamepad are just too weird. If you're using a small portable device, if you're standing and moving around while delivering a presentation (and projecting content from a computer), if you're walking around in a museum, or doing mainteinance in a plane, and seeing an augmented reality ( http://en.wikipedia.org/wiki/Augmented_reality ) overlay of information about real objects... All these are just a few of the many examples of interacting with computer systems where having traditional interface devices is just not too practical, and you go for gestures, voice or touch interfaces, for instance.

So, these are the reasons why interaction people have always loved gestures (and voice and touch). The reason why, for the rest of us, it looks like this is a recent phenomenon is that only recently technology has become cheap, small and fast enough that you can do that without a bunch of bulky sensors attached to you, without a uniformly colored background behind you, without expensive cameras, image capture card, magnetic sensors, RISC supercomputers or big clusters of computers... You get the idea.

So, people are finally able to implement for real and without an enormous budget the systems they've drooled about for years, for several good reasons, too. It makes people excited.

As for games... A lot of these excited people are also big fans of games, so they're excited to bring gestures, voice and touch to games as well. A lot of these people are also technical guys, so despite knowing these forms of interaction aren't the best for the current games (they are not dumb, after all), they figure they can create the tools, get marketing to build some hype, and trust all those creative artists and game designers out there to make games for which gestures actually make sense and are fun (and cause as few repetitive effort injuries as possible! Wiimote, I'm glaring at you!). Games are also a great way to introduce a new for of interaction to the masses (you learn new stuff a lot faster having fun with it than being forced to learn it), so it's another reason why people are pushing for gestures and other new stuff in games.

Finally, that scenario you described (arriving home late tired from work and wanting to relax for a bit) is only typical for -your- kind of player. Right now, most of us are that kind of player, too. But there's a huge market beyond us "harcore" (or old style or however you choose to call most people who play computer and console videogames today) that is pretty much untapped. Younger kids who do not work and seem to have infinite stores of energy until they flop and sleep for 10 hours. The infamous Wii soccer moms. People who only want to play as an excuse to be with and mess with their friends (think of drinking games). Older people who never grokked this whole computer thing. Younger people who think us computer users are old people, because the good stuff now is in using mobiles, not those clumsy notebooks. People who play real quick during lunch breaks or while riding the bus to school... Probably a lot more user profiles I'm not thinking of. When people invest in new kinds of interaction in games, they're not necessarily just trying to please the people they -already- are pleasing (though they will sure be happy if they manage to do that, too), but they are mostly trying to grab NEW people. Compare the number of people that play games with the number of people that do something like, say, watch TV or talk in cellphones, and you see games still have a lot of space to grow (not by chance, digital TV and cellphones are two platforms where people are trying to increase gaming, and where traditional interface devices are not so great).

Even for us that actually like current games just like they are and like to play, like you, to relax... Don't we often complain about how all games are so very similar? There's like less than half a dozen styles of games, some combinations of these styles, and within one style, all games feel pretty much like the same, aside from a few details here and there, the art frills (which are great and all, but don't -actually- change how you -play-) and maybe the story, if it's there and if it's not yet another half-***** implementation of the heroes' journey. So, even for us that like games as they are, a lot of us are beginning to feel they are getting old, and maybe new forms of interaction will lead to new forms of gameplay that will bring us some much needed fresh air in our hobby. Guess that's another reason why people hope gesture interfaces and other new forms of interaction do well with games.

As long as they remember these things are not best suited for a lot of games, especially the games we have today for the players we have today. And keep making the games with regular interfaces that we like, along with the cool new stuff.

And finally... The reasons why I, specifically, chose to do a test with a game, especially with CoH (which actually lets you at least move in 3D). It's a great way to show my system works in 3D environments (for those cases when you are walking around in a virtual or augmented environment and can't use regular devices). It's a great way to show I can integrate my system with other applications without needing to do any changes to them, or recompiling, or even letting them know I'm using gestures. It's great to show the system is stable and fast enough. It makes it easier to draw in users for testing (it looks more like having fun than like work). Gameplay mechanics in CoH put there to deal with lag (selecting targets instead of aiming, queueing powers, powers requiring longish animation times) mean that I'm not so penalized by the time it takes to perform a gesture, compared with pressing a button, for instance. It gives me kick *** screenshots and demo movies (check out those other tests in my video where I don't use CoH, or other gesture stuff in youtube, and notice how absolutely boring the stuff us technical, non-artsy people make our test applications). And last but not least... I'm an avid CoH fan, so any excuse to bring the game into my work environment brightens my day and I've had a lot of fun implementing and testing with it.