VirtuCamera plugin: Virtual camera for Houdini (Py3) and iOS

   3871   13   2
User Avatar
Member
75 posts
Joined: July 2019
Offline
This is unofficial HDA bridge for VirtuCamera iOS app (https://virtucamera.com/)
Use it to connect and record camera movement from iOS devices and send the data to cameras in Houdini

Install HDA and libraries:
Unzip VirtuCameraHoudini-2.1.1.zip archive to any folder:
Copy 'virtucamera' and 'packages' folders to houdini18.5 user preference folder (ex. C:\Users\<user_name>\Documents\houdini18.5)
Run Houdini Python 3 version (developed and tested in Houdini Indie 18.5.582 - Python 3)
Import HDA in Houdini
(File -> Import -> Houdini digital -> Select VirtuCameraHoudini-2.1.1\otls\virtucamera.2.1.1.hdalc -> Install)

How to use:
0. Create new obj camera at scene
1. Create virtucamera node at obj context (Tab - VirtuCamera)
2. Press Open Capture Window:
Attention: Do not move Capture Window manually by mouse. Use Window position and size parameters. Change those parameters and press Open Capture Window button again to reopen window in new position and size.
Explanation: It's required because current implementation uses Screenshot method to capture video from your pc screen and send it to mobile device. Houdini need to know where Capture Window on screen to send this data to iOS device.
3. Press Run Server button
4. Open QR code or skip this step and use your_pc_ip_address:port later.
5. Install VirtuCamera app on you iOS mobile device (Tested with iPhone 11): https://apps.apple.com/us/app/virtucamera-unlimited/id1461676842 [apps.apple.com]
6. Run the VirtuCamera app
7. Connect to pc by pressing Scan QR code or connect manually by your_pc_ip_address:port
8. Choose your camera (ex. /obj/cam1)
9. Tap on black screen with eye icon to see video stream from your pc.
10. Press chain button in right top corner to calibrate mobile device and send transformation data to Houdini.
11. Look around until calibration done. Now you should see that your mobile device movements are sending to camera in Houdini.
12. Press record movement button (pure red) to start recording.
13. When the recording will be done wait a minute for plugin to put the data as keyframes on your camera.

Regarding development:

Made by using open-source API - PyVirtuCamera: https://github.com/shycats [github.com]
Feel free to modify HDA and share changes at the topic

It's required to import PyVirtuCamera python 3 libraries in Houdini to use a server functionality.
This is the main reason you need to run Houdini python 3 version.
https://github.com/shycats/PyVirtuCamera/releases [github.com]
I've already downloaded and put PyVirtuCamera-cp37-windows-x64-v2.1.1.zip to the plugins archive.


Questions for community:
1. Are there any ways to optimize python keyframes set process?
def set_camera_transform_keys(self, vcserver, camera_name, keyframes, transform_matrix_values):
""" Must set keyframes on the transform of the specified camera.
The frame numbers are provided as a tuple of floats and
the transform matrixes are provided as a tuple of tuples of 16 floats
with 4x4 transform matrixes, with a matrix for every keyframe.
The first element of the 'keyframes' tuple corresponds to the first
element of the 'transform_matrix_values' tuple, the second to the second,
and so on.
* The up axis is Y+
* The order is:
(rxx, rxy, rxz, 0,
ryx, ryy, ryz, 0,
rzx, rzy, rzz, 0,
tx, ty, tz, 1)
Being 'r' rotation and 't' translation,
Is your responsability to rotate or transpose the matrixes if needed,
most 3D softwares offer fast APIs to do so.
Parameters
----------
vcserver : virtucamera.VCServer object
Instance of virtucamera.VCServer calling this method.
camera_name : str
Name of the camera to set the keyframes to.
keyframes : tuple of floats
Frame numbers to create the keyframes on.
transform_matrix_values : tuple of tuples of 16 floats
transformation matrixes to be set as keyframes on the camera 'camera_name'
"""
camera_node = hou.node(camera_name) #Get camera node
for x in range(len(keyframes)):
key = hou.Keyframe()
key.setFrame(keyframes[x])

matrix = hou.Matrix4(transform_matrix_values[x]) #Convert tuple to matrix4 format
translation = matrix.extractTranslates('srt')
rotation = matrix.extractRotates('srt')

key.setValue(translation[0])
camera_node.parm('tx').setKeyframe(key)
key.setValue(translation[1])
camera_node.parm('ty').setKeyframe(key)
key.setValue(translation[2])
camera_node.parm('tz').setKeyframe(key)

key.setValue(rotation[0])
camera_node.parm('rx').setKeyframe(key)
key.setValue(rotation[1])
camera_node.parm('ry').setKeyframe(key)
key.setValue(rotation[2])
camera_node.parm('rz').setKeyframe(key)

2. Are there any ways to capture pixel buffer from scene viewer? I would like to know, because screenshot capturing is slow and not cool, but it works
Edited by Faitel - June 4, 2021 02:00:03

Attachments:
virtucamerahoudini.png (3.1 MB)
VirtuCameraHoudini-2.1.1.zip (11.7 MB)

User Avatar
Member
12 posts
Joined: Oct. 2015
Offline
Hi Faitel, thanks for this fantastic addon.

I have this error running on Houdini

Let me know if there is anything I can help. Thanksss

VirtuCamera: QR Code generated in temp folder
VirtuCamera: Houdini plug-in version 2.1.1 | Server version 2.1.1
VirtuCamera: Accepted connection from: 192.168.2.36:54235

VirtuCamera: Client connected

VirtuCamera: Camera has changed to /obj/cam1
Exception in thread Thread-6:
Traceback (most recent call last):
File "C:\PROGRA~1\SIDEEF~1\HOUDIN~1.596\python37\lib\threading.py", line 926, in _bootstrap_inner
self.run()
File "C:/PROGRA~1/SIDEEF~1/HOUDIN~1.596/houdini/python3.7libs\hou.py", line 96303, in __threadRun
self.__run()
File "C:\PROGRA~1\SIDEEF~1\HOUDIN~1.596\python37\lib\threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "vc_core.pyx", line 2289, in virtucamera.vc_core.VCServer._handle_tcp_socket
File "vc_core.pyx", line 2144, in virtucamera.vc_core.VCServer._proccess_command
File "vc_core.pyx", line 1336, in virtucamera.vc_core.VCServer._async_callback
File "vc_core.pyx", line 1340, in virtucamera.vc_core.VCServer._async_callback
File "vc_core.pyx", line 1953, in virtucamera.vc_core.VCServer._send_scene_status
File "Object/virtucamera, PythonModule", line 297, in get_camera_focal_length
AttributeError: 'NoneType' object has no attribute 'eval'
Poly Cruncher @ Naughty Dog.
User Avatar
Member
75 posts
Joined: July 2019
Offline
Hi Santi, I'm sorry to hear that you got an error.
Plugin failed to find 'focal' parameter on your camera object, I don't know why exactly.
I've made the quick test (attached video and hip file) in Houdini 18.5.596 Py3 version, and the plugin still works for me.
Please try again and attach some screenshots or video if possible.
Edited by Faitel - June 24, 2021 08:00:03

Attachments:
virtucamera_test.hiplc (908.6 KB)
virtucamera2021-06-24 16-20-14-1.m4v (6.2 MB)

User Avatar
Member
12 posts
Joined: Oct. 2015
Offline
Thanks Faitel. I got it working now.
Poly Cruncher @ Naughty Dog.
User Avatar
Member
2 posts
Joined: March 2017
Online
Works so well, incredibly smooth compared to using in Maya.

Thanks for creating and sharing!
User Avatar
Member
3 posts
Joined: April 2022
Offline
Will this work on the houdini FX version?
Having connection problems

Attachments:
error.jpg (35.8 KB)

User Avatar
Member
75 posts
Joined: July 2019
Offline
I don't have access to Houdini FX version, but it still works with Houdini 19.0.589 Py3 Indie.

Attachments:
virtucamerah19p3.png (4.3 MB)

User Avatar
Member
3 posts
Joined: June 2019
Offline
This is incredible! Working great for me so far. Just curious if there's a way to hear audio playback on the timeline while recording. Audio works during regular playback but not during recording initiated from the app. Thanks so much for your effort on this!
User Avatar
Member
3 posts
Joined: April 2022
Offline
aew928
This is incredible! Working great for me so far. Just curious if there's a way to hear audio playback on the timeline while recording. Audio works during regular playback but not during recording initiated from the app. Thanks so much for your effort on this!
Do you run on the "Indie" version?
User Avatar
Member
3 posts
Joined: June 2019
Offline
unhrt
aew928
This is incredible! Working great for me so far. Just curious if there's a way to hear audio playback on the timeline while recording. Audio works during regular playback but not during recording initiated from the app. Thanks so much for your effort on this!
Do you run on the "Indie" version?
Yes. Using indie.
User Avatar
Member
75 posts
Joined: July 2019
Offline
aew928
This is incredible! Working great for me so far. Just curious if there's a way to hear audio playback on the timeline while recording. Audio works during regular playback but not during recording initiated from the app. Thanks so much for your effort on this!
I've added audio playback support. That's not a perfect solution because of VirtuCamera's manual frame by frame playback mechanic, but better than nothing.

VirtuCamera plugin: Virtual camera for Houdini - Update 2.1.2

  1. PyVirtuCamera Library is updated to version 2.1.2 [GitHub [github.com]]
  2. Added support for scrubbing and playback audio files

Don't forget to remove VirtuCamera folder and otls/virtucamera-2.1.1.hda file from previous version and replace with new ones.
I'll upload demo video as soon as possible.

Tested on Windows 10, Houdini 19.0.622, iPhone 11.
Edited by Faitel - May 18, 2022 20:57:51

Attachments:
VirtuCameraHoudini-2.1.2.zip (13.9 MB)

User Avatar
Member
3 posts
Joined: June 2019
Offline
Faitel
aew928
This is incredible! Working great for me so far. Just curious if there's a way to hear audio playback on the timeline while recording. Audio works during regular playback but not during recording initiated from the app. Thanks so much for your effort on this!
I've added audio playback support. That's not a perfect solution because of VirtuCamera's manual frame by frame playback mechanic, but better than nothing.

VirtuCamera plugin: Virtual camera for Houdini - Update 2.1.2

  1. PyVirtuCamera Library is updated to version 2.1.2 [GitHub [github.com]]
  2. Added support for scrubbing and playback audio files

Don't forget to remove VirtuCamera folder and otls/virtucamera-2.1.1.hda file from previous version and replace with new ones.
I'll upload demo video as soon as possible.

Tested on Windows 10, Houdini 19.0.622, iPhone 11.
You absolute beast. Its more than enough to pick up on basic audio queues for timing purposes. Fantastic job!
User Avatar
Member
46 posts
Joined: Jan. 2014
Offline
This is super cool , great job! thanks for sharing.

does it stream translation as well, like getting xforms from some AR type of camera ?

btw, regarding your question (1), use parm.setKeyframes() instead, and pass in a Keyframe() array, you'll get faster times than adding keyframes at every loop iteration, according to the docs. (see screenshot attached)
So maybe you could find a way to store all transforms during stream and set keyframes at a later stage ? (you'l need to have a frame range defined of course, as to know when its time to gather xforms and bake)
Edited by fabriciochamon - June 13, 2022 19:55:26

Attachments:
keyframes.jpg (26.9 KB)

User Avatar
Member
75 posts
Joined: July 2019
Offline
fabriciochamon
This is super cool , great job! thanks for sharing.

does it stream translation as well, like getting xforms from some AR type of camera ?

btw, regarding your question (1), use parm.setKeyframes() instead, and pass in a Keyframe() array, you'll get faster times than adding keyframes at every loop iteration, according to the docs. (see screenshot attached)
So maybe you could find a way to store all transforms during stream and set keyframes at a later stage ? (you'l need to have a frame range defined of course, as to know when its time to gather xforms and bake)

Thank you for the tip, Fabricio! I'll test it. I'm also thinking of shifting from recording in device to recording in chops as alternative mode because I see that there are a lot of other ways to utilize device xforms and maybe it isn't smart to lock everything just on python and camera. So, I am actively discovering CHOPs at the moment.

PS: Thanks for your videos
  • Quick Links