Hi,

I think I've forgotten something fundamental I know you can run frames in a single batch, but how do I run work items in a single batch?

OpenGL ROP frames render far faster the load times for Hython, even using Services, so I need to run batches of work items... but damned if I can remember how.

Simple example file attached. If I run the Rop Fetch that runs the OpenGL ROP it works correctly, but it runs each OpenGL rop in a separate Hython which is grindingly slow compared to running all the work items in a single Houdini session.

I tried InProcess Scheduler but that immediately errored out on me. Perhaps I did something wrong there too, but googling/docs didn't seem to clarify it for me

I'm using 18.5.596 on Win 10 and cannot move to 19 at this time. I could test in 18.5.773 though if that helps.

Cheers,

Peter B

Same question for the Composite ROP TOP actually... Frames for sure, but if I'm running 100 work items in "single frame" mode (and creating the frame number with, say, @wedgeindex) how do I just run them all in one batch?

Try enabling services, on the rop fetch go to the services tab and enable services, turn off 'all frames in one batch', keep
In default mode the fetch takes about 10 seconds to start, and 0.2 seconds per work item. After doing the above, startup time is almost instant, and cook time is still 0.2 seconds per item, so its pretty quick.

My limited understanding of whats going on here:

By default the scheduler is spinning up a hython per workitem, so if your houdini takes 10 seconds to start, so does each workitem. Using services will spin up a hython (or hythons) in the background that stay active and ready to load the temp hips that tops creates per workitem. You skip the startup time, hence zippier performance.

Seperate to that is the idea of running the workitems in parallel, or batching them to run as a single process, which can be enabled with the 'all frames in one batch' toggle. If you enable this, stuff starts up very slowly, which makes no sense until you look at the warnings. Annoyingly you can't use batch and services together, so it defaults to spinning up a hython, hence the delay to start.

Fastest still would be to run it all in the current session, but as you've found out, the in process scheduler doesn't work with almost all tops nodes, annoying. Maybe in a future release...

Recorded a little demo if it makes things clearer. Hopefully your studio is services friendly, otherwise this is all moot!

https://youtu.be/I7GPZxygUys [youtu.be]

(edit)

Ah, I think there were tops improvements made in 19 too, so generally stuff moves faster than in 18.5. An extra incentive to update maybe?

Cheers,

-matt

Thanks Matt, I actually started with Services but it didn't really help, I did discover the warning/error about Batch mode not working in Services eventually

I'll try Services again maybe I had some combo of things not working. Loading the .hip file is pretty slow for me because pipeline, though it's faster than loading the whole Hython + HIP file, so in theory Services should offer a speed up.

One very large problem is Services don't clear the memory used after the cook, SESI have acknowledged this as an existing RFE, so for some things I can't use Services because then the memory on my machine gets consumed until I exit the services which isn't ideal. However, for the GL/Comp stuff mentioned here, memory isn't a problem so I hopefully could use the services just for those bits. If it works that is

Sadly can't move to 19 for a while, maybe in 2022 hopefully

Appreciate you taking the time!

Cheers,

Peter B

So this works wonderfully in my test case, but of course in my prod file, it doesn't work Alas.

I'm guessing from the lack of reply that "group the work items into a single cook" isn't a thing?

Cheers,
Peter B

In your original .hip file, the problem with batching for the OpenGL ROP is that the input work items all have the frame range (1, 1, 1). In order for the second ROP Fetch to generate a batch, it needs to know what the frame range boundaries are on the input work items. What happens now is that it assumes that each input item is from a distinct sequence, and therefore ends up creating 10 batches each with a single frame and a range of (1,1,1). Batching itself only works over a range of frames, because a "batch" is just a work item that calls rop.render(...) over with a range instead of a single frame.

One way to make the example file work is to enable batching on your ROP Geometry, and enable the "Automatically Set Missing Frames" toggle. Since the incoming wedges have no frame information, the ROP Geometry will fallback to assuming the desire frame range is (1, num_items, 1). You'll then end up with 10 work items with frames ranging from 1 to 10. Enabling batching on the downstream ROP Fetch will then just work because the incoming items have correct frames/ranges set on them.

If you want to avoid modifying the ROP Geometry node, you can also drop an Attribute Create in between the ROP Geometry and ROP Fetch, and use it to manually update the "range" attribute to (1,10,1). That way that the ROP Fetch will once again treat the input work items as a sequence of 10 frames, instead of 10 sequences of 1 frame each. You can enable batching on the ROP Fetch without any other changes to that node.

A third option is to put a Wait for All after the Rop Geometry and then wire the second ROP Fetch into that with the frame range start/ends parms set to 1, @partitionsize and batching enabled.

In general, the ROP Fetch and in particular batching is very tied to frame ranges. There's an existing RFE to add more options for batching, e.g. by attribute or by fixed numbers of work items. In that case, the script cooking the ROP would have to manually run a loop and call rop.render(frame=1) for each entry in the batch, instead of invoking the ROP once over a frame range. That would also solve this issue -- we can look into adding those controls over batching sooner rather than later.