Is there a better way to pause and resume caching? Let's say I have 240 frames. I keep working on something else in the background while caching keeps happening. I realize that 170 frames have been cached and now I need to pause it to keep working on something else so that I have enough CPU power and RAM.
Hours pass by and I want to close the day, shutdown my machine and go to sleep.
Next morning I wake up, I want to start caching at 171st frame onwards.
What's the ideal way?
Specifically asking for bgeo.sc type caching, not for .sim caching.
A smarter way to pause and resume caching?
2667 8 0-
- proceduralist
- Member
- 206 posts
- Joined: 5月 2021
- Offline
-
- Enivob
- Member
- 2637 posts
- Joined: 6月 2008
- Offline
-
- proceduralist
- Member
- 206 posts
- Joined: 5月 2021
- Offline
-
- icecreamumai
- Member
- 65 posts
- Joined: 3月 2012
- Offline
How about having Thinkbox's Deadline Submitter(free) on your machine? so you can schedule simulation and rendering scenes.
You can also add dependencies. After sim scene has been finished, the rendering scenes are automatically started even midnight.
if the simulation is still on-going during your working time, set houdini's CPU low-usage(maybe half usage) from Task Manager.
if you pause the simulation, you need to restart the sim, so the idea is keep the simulation but make CPU low usage.
After you finished working, you can set it back to normal.
If you have a lot of memory, you can set concurrent task from 1 to 3. meaning you can render 3 machines at same time.
Sometimes this boost your rendering time
You can also add dependencies. After sim scene has been finished, the rendering scenes are automatically started even midnight.
if the simulation is still on-going during your working time, set houdini's CPU low-usage(maybe half usage) from Task Manager.
if you pause the simulation, you need to restart the sim, so the idea is keep the simulation but make CPU low usage.
After you finished working, you can set it back to normal.
If you have a lot of memory, you can set concurrent task from 1 to 3. meaning you can render 3 machines at same time.
Sometimes this boost your rendering time

Edited by icecreamumai - 2023年7月8日 07:40:03
-
- proceduralist
- Member
- 206 posts
- Joined: 5月 2021
- Offline
-
- tamte
- Member
- 9118 posts
- Joined: 7月 2007
- Online
The only way I know to resume sim after the crash or deliberate kill on the farm is to cache checkpoint files alongside
https://www.sidefx.com/docs/houdini/dyno/cache.html#explicit-dop-disk-caching [www.sidefx.com]
You can play with Checkpoint Trail Length and Checkpoint Interval to save only small amount of trailing .sim filés exactly for the purpose of resuming
While you still filter down the sim data and save normally as bgeosc or other format
https://www.sidefx.com/docs/houdini/dyno/cache.html#explicit-dop-disk-caching [www.sidefx.com]
You can play with Checkpoint Trail Length and Checkpoint Interval to save only small amount of trailing .sim filés exactly for the purpose of resuming
While you still filter down the sim data and save normally as bgeosc or other format
Edited by tamte - 2023年7月8日 11:52:47
Tomas Slancik
CG Supervisor
Framestore, NY
CG Supervisor
Framestore, NY
-
- proceduralist
- Member
- 206 posts
- Joined: 5月 2021
- Offline
-
- tamte
- Member
- 9118 posts
- Joined: 7月 2007
- Online
from the description of the Deadline approach it looks like it will still need to keep the sim fully in memory and can potentially just free up some CPU resources, so for heavy sims that may be a no go, unless you just need to perform low memory tasks while the sim is on low CPU
On top of that it doesn't protect you from crashes
so checkpoints may be a safer way and also the only way I know that should allow you to truly resume the sim, even though I didn't have a need to use them in production, so not speaking from experience, but just in how I understand it should work
the .sim files may be heavy, but you may need to keep only 1 last rolling frame on disk at a time so it may be worth for long sims
and you can always combine the techniques, that way you can use deadline or other scheduler if it allows you to decrease the resources on the fly but you can also kill the job if needed if you are checkpointing and then submit from that frame later
On top of that it doesn't protect you from crashes
so checkpoints may be a safer way and also the only way I know that should allow you to truly resume the sim, even though I didn't have a need to use them in production, so not speaking from experience, but just in how I understand it should work
the .sim files may be heavy, but you may need to keep only 1 last rolling frame on disk at a time so it may be worth for long sims
and you can always combine the techniques, that way you can use deadline or other scheduler if it allows you to decrease the resources on the fly but you can also kill the job if needed if you are checkpointing and then submit from that frame later
Edited by tamte - 2023年7月10日 12:18:33
Tomas Slancik
CG Supervisor
Framestore, NY
CG Supervisor
Framestore, NY
-
- Soothsayer
- Member
- 874 posts
- Joined: 10月 2008
- Online
I find that sim workflows with checkpoints are very easy to screw up. It's a bit like takes, it sounds great at first and then things go wrong and you don't notice until it's too late. That's just my personal experience and it may work well for some.
On Linux you can also put your machine to sleep and the just awake it at a later point and things will continue where they stopped. Not 100% sure that works with opencl on GPUs.
On Linux you can also put your machine to sleep and the just awake it at a later point and things will continue where they stopped. Not 100% sure that works with opencl on GPUs.
Edited by Soothsayer - 2023年7月10日 12:33:34
--
Jobless
Jobless
-
- Quick Links