Deadline concurrent tasks not working if single batch cooks

   1379   2   2
User Avatar
Member
2 posts
Joined: Jan. 2020
Offline
I have GPU affinity and 2 concurrent tasks working correctly in TOPS with the Deadline scheduler. However, I’ve run into this problem: if the preceding TOPS node completes enough frames to make the first batch for the next node (using Deadline scheduler set to 2 concurrent tasks), a Deadline job is properly submitted, but Deadline only runs one task at a time. The Deadline queue continues to populate, however only one task is ever run at a time.

In contrast, if at least two batches of frames have already been completed, then Deadline runs 2 concurrent tasks as it should. Subsequent jobs in the queue are also run concurrently.
User Avatar
Member
7024 posts
Joined: July 2005
Offline
I'm experiencing something similar, I think, though I could be wrong!

Further complicating it, I have two Deadline schedulers so that I can use different Groups.

I definitely get concurrent jobs running on a single worker for most of the TOP nodes, but some steadfastly refuse to run concurrent jobs even though there are many frames that could run at the same time.

I have a Wait for All before the "problem" TOP node, hoping that would make it work, but it doesn't seem to help...

SOLVED (maybe): Looks like you need to set Generate When to "When All Upstream Items Cooked". Not sure if that reduces efficiency though. In my case I'm using Generation IFD TOP followed by Render IFD. I have to set Render IFD to Generate When to "When All Upstream Items Cooked" and then I get concurrent tasks working well.

I also have the Generate IFD set to "All Frames in 1 batch" because IFDs should always gen very fast Given that, efficiency should be OK-ish but for a longer IFD gen, not sure where the sweet spot lies.

Cheers,

Peter B
User Avatar
Member
7024 posts
Joined: July 2005
Offline
So I just discovered that if you modify the concurrent tasks after a job has been submitted to Deadline, using the Deadline monitor, it does _not_ recognize that change. So this might be more of a Deadline issue? I'm speculating however.
  • Quick Links