Mantra startup time + PDG - [was Mantra geometry preparation]

   668   18   3
User Avatar
Member
3908 posts
Joined: June 2012
Offline
Is there any way to use PDG to improve the bottleneck in preparing geometry for rendering in Mantra?

Currently a very light-weight scene, i.e. a particle system or simple geo, spends most of the ‘render time’ in geometry preparation.

Thx!
Edited by goat - March 18, 2019 01:31:57
User Avatar
Member
3908 posts
Joined: June 2012
Offline
@Daryl Thanks for the question.

If anyone can help out with an answer to the original question I'd appreciate that.
User Avatar
Member
3908 posts
Joined: June 2012
Offline
To clarify the question for anyone whom may wish to help; this is not a SOPs, DOPs, or ROPs question but a Mantra + PDG question. The geometry setup internally with Mantra is a huge delay when doing lightweight renders, if PDG can help that part I'm all ears.
User Avatar
Member
9572 posts
Joined: July 2005
Offline
goat
To clarify the question for anyone whom may wish to help; this is not a SOPs, DOPs, or ROPs question but a Mantra + PDG question. The geometry setup internally with Mantra is a huge delay when doing lightweight renders, if PDG can help that part I'm all ears.

Do you feel that delay with rendering with IPR?
jason iversen, houdini procedural pipeline supervisor @ weta digital
also, http://www.odforce.net [www.odforce.net]
User Avatar
Member
3908 posts
Joined: June 2012
Offline
jason_iversen
Do you feel that delay with rendering with IPR?


IPR is perfect! No delay and I was hoping that PDG would allow us to access the magic IPR magic in a render to disk/flipbook
User Avatar
Member
9572 posts
Joined: July 2005
Offline
goat
jason_iversen
Do you feel that delay with rendering with IPR?
IPR is perfect! No delay and I was hoping that PDG would allow us to access the magic IPR magic in a render to disk/flipbook

Then I suspect you're not suffering from a computational bottleneck at all but rather relatively slow launch of mantra and mplay. Are you on Windows? Linux?
jason iversen, houdini procedural pipeline supervisor @ weta digital
also, http://www.odforce.net [www.odforce.net]
User Avatar
Member
3908 posts
Joined: June 2012
Offline
sure - it's the startup and initial setup of mantra. It occurs on Linux, windows and mac os. Do we have a solution for this slowness?
User Avatar
Staff
446 posts
Joined: Sept. 2012
Offline
If it's within Mantra itself, I'm afraid it won't be able to take advantage of PDG, at least for now. Any change to make Mantra faster would need code changes within Mantra, and to this point we have not done so in the case of PDG. I'll pass along the query though to our Mantra team, so they may take this into consideration.
- Ken Xu
User Avatar
Staff
2082 posts
Joined: July 2005
Offline
goat
sure - it's the startup and initial setup of mantra. It occurs on Linux, windows and mac os. Do we have a solution for this slowness?
Depending on your studio's setup, mantra's startup time can be affected by engine mode. What times do you get when you run:
% time mantra -e none < /dev/null
% time mantra -e basic < /dev/null
% time mantra -e full < /dev/null
On my machine, the startup time difference is 0.5s vs. 0.9s. You can also control this using the
MANTRA_ENGINE_PROCEDURAL variable.
User Avatar
Member
3908 posts
Joined: June 2012
Offline
Testing those I get similar times, but running a flipbook on a torus it takes ~3 sec between frames. If launch times are fast then what is this disparity and can it be reduced to 0.59 sec instead of 3 sec? thanks!

mantra -e none
real 0m0.599s

mantra -e basic
real 0m0.726s

mantra -e full
real 0m0.722s
User Avatar
Member
9572 posts
Joined: July 2005
Offline
It would be nice if you could use IPR mode for a flipbook, but alas you can't - not unless you write a script to step through the timeline yourself and rip the image from hou.IPRViewer and append it to a flipbook at timed intervals, checking for a time limit or not hou.IPRViewer.isRendering().

… which brings up a little point: the Mantra “Render Time Limit” feature seems to exit with an error code when it shouldn't.
jason iversen, houdini procedural pipeline supervisor @ weta digital
also, http://www.odforce.net [www.odforce.net]
User Avatar
Member
3908 posts
Joined: June 2012
Offline
Thanks Jason - I guess in trying to see if PDG can help with this issue we need to understand what is causing the 3 sec delay between frames. If it's not geometry preparation, as IPR is instant, and it's not launch times of Mantra, as that is only 0.5 sec, do we know what is causing the missing 2.5 sec?
User Avatar
Member
9572 posts
Joined: July 2005
Offline
Make friends with vtune or strace is what I do.

Often it's loading of libraries and stuff.

Are you on Linux, Mac or Windows? Are you running in a deeply stacked production environment?
jason iversen, houdini procedural pipeline supervisor @ weta digital
also, http://www.odforce.net [www.odforce.net]
User Avatar
Member
3908 posts
Joined: June 2012
Offline
With more testing it looks like the culprit has been found.

The ‘missing’ 2.5 secs appears to be not geometry or launch times but material setup. Putting a simple shader like constant, specular or a texture removes the delay, down from ~4 sec a frame to 1 sec. The rendering time part is the same, just the delay between frames is removed.

Without any geometry in the scene, frames launch straight away one after each other, whilst adding a torus will make each frame take a total of ~4 sec. We assume that it's using the default plastic shader. Appling constantVop material reduces the total rendering time for a frame to 1 sec.

My assumption is that LOPs could fix this and there is probably very little PDG can do to improve material setup time.
User Avatar
Member
9572 posts
Joined: July 2005
Offline
Ah yes, okay. Grubbing around the back of my memory, I thought there was a VEX (disk) cache that was persistent and able to be reused render to render… something which - at least for a while - was only enabled on Linux? Or perhaps it invalidates that cache frame to frame? SideFX might have to jump in here and jog my memory.

Usually settings for these caches turn up in the environment variables (http://www.sidefx.com/docs/houdini/ref/env.html) - like HOUDINI_DISABLE_VEX_INSTANCE_CACHE
Edited by jason_iversen - March 19, 2019 14:41:08
jason iversen, houdini procedural pipeline supervisor @ weta digital
also, http://www.odforce.net [www.odforce.net]
User Avatar
Member
3908 posts
Joined: June 2012
Offline
oh cool! That's great to hear.

I am getting some permission errors in the 17.5, i.e.

Error:       Saving to stream failed (ref: String Save <Permission denied> ).

but it more sounds like a bug that VEX shaders aren't cached. I couldn't see any env variable that was relevant unfortunately.

I'll submit a bug! bug #95480
Edited by goat - March 19, 2019 19:04:48
User Avatar
Member
22 posts
Joined: Jan. 2015
Offline
I have run into this too. Using a ropmantra node in TOPS to render 100 frames of a very simple Font SOP text node take around 48 seconds a frame - even with “All frames in batch” turned on. Whereas a standard Mantra node in Output context takes 12 seconds per frame.
User Avatar
Member
3908 posts
Joined: June 2012
Offline
morganv123
I have run into this too. Using a ropmantra node in TOPS to render 100 frames of a very simple Font SOP text node take around 48 seconds a frame - even with “All frames in batch” turned on. Whereas a standard Mantra node in Output context takes 12 seconds per frame.

This sounds more to do with the number of CPUs in the local scheduler being set to 0, that then means the logical CPU are divide by 4. @kenxu do we have any enlightenment on why the 1/4 was used?


@morgav123 Could you try setting the CPUs to something like -1 or 128. Also try to putting 1 in the Frames per Batch.
User Avatar
Staff
446 posts
Joined: Sept. 2012
Offline
The 1/4 was used as a heuristic to avoid overloading the machine by default. We had cases where the user tried to run way too much on a single machine: 80 sims each with 250 frames with each frame baking out 1 gig of data. To top it off the sims used opencl, which drained the graphics memory - drivers do not behave well under those conditions. To prevent situations like that, we opted for some very conservative settings. Feel free to up the number of CPUs used - just please keep in mind what you're asking the machine to do
- Ken Xu
  • Quick Links