Fabricio Chamon

It was a bit of a struggle to record the motion capture and playing back live the body animation, so that body and hands match time wise

I was anticipating this, but haven't thought of a solution yet. If you eventually come up with ideas, let me know.

Maybe constraint hand and a camera to the mocap wrist, so you can see it in place?

---

Speaking of body, I've also touched MediaPipe Body in VML Streamer, to make it support world transforms as well. (previously it only had hips fixed at the origin). I can now record jumps (although with some Z jitter). Also added support for more than 1 body. (I've been able to capture up to 4-5 bodies, as soon as they don't overlap and camera is fixed/has decent lighting).

luisguggenberger
Yes it's certainly a good feature to have, been able to still use the mouse. When others in my studio tested it, they felt it was a bit unintuitive, that the webcam image doesn't behave like a mirror. When one hand accidentally got out of frame they moved the wrong hand first to bring it back into frame. (hope that discription makes sense)

If it would be me, I would probably have either two toggles (flip image/flip hands) or have the mirrored image as the default without actually mirroring the hands.

That said it's an easy fix inside Houdini for anyone who wants a different behavior.

I made a first test today adding finger motion to a body mocap and it worked out pretty good! It still needs cleanup for final animation but for previs it does a great job. It was a bit of a struggle to record the motion capture and playing back live the body animation, so that body and hands match time wise. I don't quite understand the "Time Dependant" toggle on Mocap Stream. It helps in terms of performance when the real time toggle is switched on, but for some reason I had to timeshift to get the finger animation into place.

(New release with all updates below: VisionML 1.0.1 [github.com])

Good suggestion, thanks Luis. I've removed the default image flip from vml streamer hands stream and added a preferred orientation mode in the hands decoder HDA. We now have:

First person (palms facing Z-) or
Mirror video (palms facing Z+).

The hand sides now align with real world, so it should be more user friendly. I still left "first person" as the default choice so I don't break previous hips (hopefully, haven't checked!).

---

Other than that I've re-worked on the pose compare hda to be more robust. It now considers hand local space, joint angles(more accurate) and supports point weights. You can use the weight attribute to, for example, give more "comparision" priority to a single finger or a couple of joints, disregarding other areas where you don't need to match as much. This will alleviate jitter errors and improved the pose matching on my tests.


There's new sample scene to demo the pose compare HDA changes, running inside a solver to create a minimal drawing "app". Used android sensors so my phone can control brush size (rotating front/back), which is also a nice example of multiple streams being used together:

luisguggenberger
One thing I realized using "Flip image" in VML Streamer it also flips the hand assignment. So when flipped I'm controlling the right hand using my left hand. It's an easy fix in houdini obviously, just realized this behavior changed from the previous version and was wondering if it was intentional.

This is intentional, "flip image" changing hand assignments helps making 2 hand motions quickly accessible with a single hand, so you don't have to let go of the mouse.

At least that was my experience when using the tool, but of course I'm open for suggestions. I can add a toogle on the HDA(or streamer) to prevent the behavior, what do you think ?