In Houdini 14 I used to enable the Save Checkpoints option box to be able to resume a flip simulation, now it doesn't work.

Now the output DOP seems faster but I can't figure out how to resume a sim.

Any help ?

Of course I'm setting the Cache Memory to 0 to maximize ultimate particle count. Leaving it to 5000 works as expected but only for toy sims.

Can anyone please test a simple sim ?

Why not

I'm still trying to figure this one out as well. I can't find any resources and the documentation is not clear to me. It looks like Houdini 15.5 changed its process on how to go about doing this. If you find out please let me know.

Cheers

Same here, Houdini 16.5.571, trying to resume a Sim that i ran last night. Out of 3600 frames (ouch) only 3011 got simulated. I cached that with a File Cache node, and instead of clicking the “save to disk” and let it do its thing in the background… i just put the file cache node on “write” and let my timeline “play once”. Not so much cause i'm an idiot but because everyonce in a while i'd come have a look at what's going on in the viewport to see if my velocity still had influence on the surface of the flip tank. Anyway… i believe one of my toddlers messed around on my desk early this morning and must have banged the keyboard or maybe just tried to go on youtube to watch tchootchoos and Firetrucks… :$ But one thing leading to an other it interrupted the sim (timeline playback) and i lost the dopnet cached frames and all i have left is the “.bgeo.sc” cached frames to dream of resuming the last 499 missing frames. Is that possible with bgeo.sc? or can this only work with “.sim” or checkpoints?

I don't mind running it again tonight, but it'd be sweet if i could just resume from where i stand. Any insights more than appreciated.

Thanks in advance,

Cheers,

A.

I usually just look in my directory to see how far it got…then delete the last file for good measure…and reset the range on the file cache to pick up where I last left off. Same approach with geo caching and renders.

BabaJ
I usually just look in my directory to see how far it got…then delete the last file for good measure…and reset the range on the file cache to pick up where I last left off. Same approach with geo caching and renders.

But this will not take into count the current state of the cached sim. How would i tell the dopnet to look into my cached bgeo.sc sequence and grab the last frame as initial state? Cause following your method will just start the sim again from the new start frame, which would mean in my case that the flip tank will be fresh clean again without the 3000+ frames of veolicities applied to it, no? (my dopnet cache memory isn't set to anything that would come remotely close to hold the 3011 frames… so that's all gone by now).

Cheers,

A.

PS: initial state only seem to point at “.sim” files… and i saved out “.bgeo.sc”.

Any chance i can convert a “bgeo.sc” file into “.sim” to create an initial state? or would it be missing some required info / attribute to resume from it then?

Not sure what you mean…I thought you were talking about picking up where you last left off.

If I set my save to disk file cache to Frame 1 to Frame 10,000 and run it overnight;

Come back the next morning and see it only did up to 5,000;

Then I simply set my file cache to start at Frame 5,001 and end at Frame 10,000.

It won't re-save out frames 1 to 5,000 but it will re-calculate what it need to start at frame 5,001. And yes this will take time.

I hope I didn't give the impression somehow you could sidestep the previous necessary sim calculations.

How would i tell the dopnet to look into my cached bgeo.sc sequence and grab the last frame as initial state?

but the last frame is the result of those calculations(previous frames) not the calculations themselves;

I think to do what you are suggesting would mean embedding extra dop state data into each frame so that the dop net could be initialized to pick up at whatever frame one desires, instead of re-calculating to get to that same state.

It would probably balloon the size of the files too.

But I really could be wrong about this, not going to pretend this is how it is. Maybe someone can chime in and give a better explanation?

BabaJ
Not sure what you mean…I thought you were talking about picking up where you last left off.

If I set my save to disk file cache to Frame 1 to Frame 10,000 and run it overnight;

Come back the next morning and see it only did up to 5,000;

Then I simply set my file cache to start at Frame 5,001 and end at Frame 10,000.

It won't re-save out frames 1 to 5,000 but it will re-calculate what it need to start at frame 5,001. And yes this will take time.

I hope I didn't give the impression somehow you could sidestep the previous necessary sim calculations.

How would i tell the dopnet to look into my cached bgeo.sc sequence and grab the last frame as initial state?

but the last frame is the result of those calculations(previous frames) not the calculations themselves;

I think to do what you are suggesting would mean embedding extra dop state data into each frame so that the dop net could be initialized to pick up at whatever frame one desires, instead of re-calculating to get to that same state.

It would probably balloon the size of the files too.

But I really could be wrong about this, not going to pretend this is how it is. Maybe someone can chime in and give a better explanation?

I think you're right, also why i choose bgeo.sc file cache out of te dopnet, i can't allocate more storage than i already do. “.sim” files are just way too large, i'm already at 3000+ x 200MB (per simulated frame with bgeo.sc), that's…. a lot for me for such long sequence.

Thanks for the help, i'll just resim tonight, no way i'm waiting hours now to catch up with the 3000 frames of sim i need to let the dopnet resume from where i stand.

Cheers,

A.

So i restarted the whole sim last night, and this time with checkpoints, and it is indeed slower and much greedier in storage. Still not finished after 11 hours while previously without checkpoints it did reach the 3000+ frames in about 6-7 hours. I guess it's kind of the catch to get more flexibility / safety net margin.

Thanks again!

Cheers,

A.

BabaJ
It won't re-save out frames 1 to 5,000 but it will re-calculate what it need to start at frame 5,001. And yes this will take time.

how is this behaviour even remotely helpful?
In the video tutorial SideFX produced on caching, one of the last thing she says is: with this method, you can pick up where you left off.
I wouldn't call “recalculating 3000 frames, ”Picking up where you left off".

Am I missing something? cuz this seems like a HUGE hole in the user's ability to create even mildly complex/hirez sims.

If the DOP can pass the states of individual border particles into adjacent wedges, it seems like it can pass the contents of the state of the particles (or volumes) into the next frame.

There is a massive lack of documentation on this.

T

how is this behaviour even remotely helpful?

It is helpful if one understands the meaning of it.

It all dependend on the process…some might be able to restart at a specific point - one that does not require previous calculations to be re-set up in memory.

…it seems like it can pass the contents of the state of the particles (or volumes) into the next frame…

It's not just the state of the particles, but the state of the process that gives the state of the particles.

Say some process uses many different functions in combination with each other on a per frame basis.

Those functions will have a number of variables within themselves and ones created between them to create the currnet state.

And on each frame those values of the variables are different, which results in the specific result of the ‘state of particles’ for that frame.

You can't just re-start at some point in the loop without knowing what those temp variable values are so that the process continues as it would have originally.

Except for cases in which the current ‘state of particles’ is all that is needed. Meaning, the nature of the particular process does not need to ‘carry over’ information other than the ‘state of particles’ from frame to frame.

So, it all depends.

subscriptions

BabaJ
It won't re-save out frames 1 to 5,000 but it will re-calculate what it need to start at frame 5,001. And yes this will take time.

how is this behaviour even remotely helpful?
In the video tutorial SideFX produced on caching, one of the last thing she says is: with this method, you can pick up where you left off.
I wouldn't call “recalculating 3000 frames, ”Picking up where you left off".

Am I missing something? cuz this seems like a HUGE hole in the user's ability to create even mildly complex/hirez sims.

If the DOP can pass the states of individual border particles into adjacent wedges, it seems like it can pass the contents of the state of the particles (or volumes) into the next frame.

There is a massive lack of documentation on this.

T

I think what Babaj meant is assuming that your last cached frames, depending on the size of your dopnet cache, so mabye 4950 up to 5000, are still in the cache. Then you can just pick it up from there.

My issue that houdini crashed, or let's say freaking windows 10 updates restarted my computer during the process. So in my case my cache was gone, and indeed would involve recalculating it all.

I've learned my lesson, and since i'm saving “checkpoints” every 100-250 frames to limit the damage when shit hits the fan, and it has already paid off since my last post. I got confused a bit by where to pick up the sim again with the checkpoint/initial state, because each time to reset the start frame of your sim it also resets the name of your checkpoint. What was frame 5000 after picking it up the next checkpoint is called hip.100.sim. And you'd have to re-sim from 5001, cause if like me you make the mistake to restart it at the same frame as your initial state you'd delete the previous cached frame and sent backward in time to the previous checkpoint (Grrrr). So i don't think i'm super clear here… but i agree, some more detailed examples in the doc / help would be have been appreciated indeed.

Cheers,

A.

EDIT: Seems Babaj is fast as the lightning and jumped in already. So never mind me

hmm.
trying to wrap my head around this.


It seems like you're saying: the checkpoint lets you pick up where you left off, but the “Save to Disk” option on the Output node doesn't… well it kinda does, cept you have recalculate all of the frames up to that point. so it takes the same amount of time as just starting over.



With Checkpointing method, are you saying the checkpoint contains the entire cache up that point? or does it just save the state at that frame?

in other words: if you play the sim on the timeline with checkpointing ON and you have a checkpoint on frame 50, your machine crashes while playing 51-100, you can still render frames 1-50 without recalc'ing the sim?
even though you didn't save frames 1-49 to disk in anyway?

can you use the Save to Disk In Background option in conjunction with the checkpoints?
so you'd save frame 1-50 using the output node, then you could save 51-100 after if finishes cuz you'd have a check point at 50 to start from?

Why does the Allow Caching to Disk option grey out when you click save checkpoints?

are there any tutorials on the practical implementations of this hiding somewhere?


T

ok, so, I'll start answering my own questions.

The Save to Disk and Checkpoints don't seem to play together at all.

Checkpoints are only the state of the DOP at the current frame. If you close the .hip and reopen, that means recalc'ing everything that wasn't checkpointed. However, if you click ahead in the timeline, it only calculations to that spot from the last checkpoint.

subscriptions
hmm.
trying to wrap my head around this.


It seems like you're saying: the checkpoint lets you pick up where you left off, but the “Save to Disk” option on the Output node doesn't… well it kinda does, cept you have recalculate all of the frames up to that point. so it takes the same amount of time as just starting over.



With Checkpointing method, are you saying the checkpoint contains the entire cache up that point? or does it just save the state at that frame?

in other words: if you play the sim on the timeline with checkpointing ON and you have a checkpoint on frame 50, your machine crashes while playing 51-100, you can still render frames 1-50 without recalc'ing the sim?
even though you didn't save frames 1-49 to disk in anyway?

can you use the Save to Disk In Background option in conjunction with the checkpoints?
so you'd save frame 1-50 using the output node, then you could save 51-100 after if finishes cuz you'd have a check point at 50 to start from?

Why does the Allow Caching to Disk option grey out when you click save checkpoints?

are there any tutorials on the practical implementations of this hiding somewhere?


T

Not sure if this is addressed to me. Getting a little confused and sometimes off topic myself. But… i forgot to mention something highly relevant in my workflow, my bad… it's that i have an external dop import in an other geo with a file cache node that saves out whatever is sim-ed into a separate cache. For that one i use bgeo.sc cause it's less greedy in storage. And i only use the .sim for the checkpoints. So yes, when it crashes at frame 51 and i have a checkpoint at frame 50 i don't mind all that much whether or not my dopnet cache is still there or gone, cause i rely on my file cache node from the external geo with the dopnet import. Usually when i sim stuff i even hide the dopnet and go straight into my external geo and use the “save to disk” caching there.

Cheers,

A.

oh. interesting. So ‘caching’ the DOP Import saves every frame to disk AND does so in a way that triggers the checkpointing to happen?
Or…
You setup the checkpointing, and click at the end of the timeline to force it to cache the timeline, then when if finishes you write out the bgeo files?

guess my only complaint about that method is that this doesn't seem to happen without loosing the GUI interactivity (as opposed to “write to disk in background”)

Thx for chatting with me, BTW! Much appreciated.


T

subscriptions
oh. interesting. So ‘caching’ the DOP Import saves every frame to disk AND does so in a way that triggers the checkpointing to happen?
Or…
You setup the checkpointing, and click at the end of the timeline to force it to cache the timeline, then when if finishes you write out the bgeo files?

guess my only complaint about that method is that this doesn't seem to happen without loosing the GUI interactivity (as opposed to “write to disk in background”)

Thx for chatting with me, BTW! Much appreciated.


T

Yes, you can set your file cache node to “writing” and press play or go straight to the last frame, but i just set it to “read” and load from disk and then press “save to disk”. I don't know for sure but i believe the later prevents wasting time with vierwport display and other viewport related matters, so it's usually a little faster. Also i've had the issue of just pressing play while the timeline looping option was on and it ended up in never ending caching / writing of my file cache node (unless Houdin is smart enough to see that the timeline is all simed and stop writing then unless it's a new sim, again, i'm lacking detailed examples in the doc / help to figure out what the best workflow. I feel watching tutorials, good ones, is key for FLIP/POPS/PYRO, but i'm lazy ). So “play once in that case scenario is recommend if you step away from your computer.

I'm all ears to learn better workflows, but for me this works. And yes you ineed have to manually set the checkpoints feature on your dopnet ”cache" tab, and that should be .sim files and it's really up to you how many steps you chose. i usually set a checkpoint at every 10 to 25% of my sequence length.

Cheers,

A.