Delete all sources after creating a MP4

   321   8   1
User Avatar
Member
24 posts
Joined: May 2015
Offline
Hey all,

Something I've been trying to do with no luck so far is deleting all my source files at the end of a TOP simple network.
Think ROPing a mesh, then capturing some frames, making a mosaic, a MP4, and then deleting the bgeos as well as the jpgs from the capture and the mosaic.

Issue is, while I am perfectly able to do everything down to making the final MP4, I just don't know how to tell PDG to then delete the files computed at the previous steps.



Any suggestions ?
Edited by Wolrajh - March 17, 2019 14:07:24

Attachments:
TopRemove.JPG (28.0 KB)

User Avatar
Staff
443 posts
Joined: Sept. 2012
Offline
If you right click, there is “Delete Selected Node Results From Disk”. However, this will dirty the node and all downstream nodes too. There isn't a way to blow away the cache and somehow not affect the state of the workitems - and it would be incorrect to do so, because once the cache is destroyed the bake results that the workitems point to is invalid, and if you try to recook downstream nodes it will fail.
- Ken Xu
User Avatar
Member
24 posts
Joined: May 2015
Offline
Hey,

Thank for your answer, sadly that's not really what I meant

What I want is basically to automatically cleanup the various step results once the final composite has been output. Obviously deleting the files before getting to the next step would be pointless
I just can't imagine there isn't a way to do so. Have another branch with all the outputs waiting for a go from the main branch ? Carry over a list of all the files that have been output up until now ? In both case, I just don't know how to do it…
User Avatar
Staff
443 posts
Joined: Sept. 2012
Offline
Ok, in that case maybe have a final step to copy your final file somewhere else, the RMB at the root of the PDG graph, and choose “Delete Selected Node Results From Disk”. This should get rid all files that have been produced.
- Ken Xu
User Avatar
Staff
141 posts
Joined: May 2014
Offline
One option that would work out-of-the-box is to write all of the files you want to discard to a different directory, and have the Remove File node remove that directory. PDG already creates as session-specific temporary directory which can be accessed with the $PDG_TEMP variable - this is where log files and scripts created by PDG are written, among other things. Scheduler nodes have a right mouse button entry for deleting that directory manually, but $PDG_TEMP can also be used as the path in a Remove File node. You could of course use your own though, e.g. something like $HIP/tempfiles/…

Alternatively, something like the following would also work:


I have two wait for all nodes - the first one waits for the work items that I want to keep, and the second one has the nodes that produced intermediate files that should be deleted. The attribute delete after the first wait for all is there to delete the Output attribute. Then, there's a third wait for all which depends on both branches, and a Remove File configured to delete all of the files on the work item in the third wait for all. This setup ensures that the delete step isn't run until after everything else is cooked, and the delete step will also not touch any of the files from the left branch.

There are pros and cons to both approaches. If your work items produce files that aren't reported to PDG, the Remove File node will never find out about them so they won't be cleaned up. In that case using a pre-determined temporary directory is more reliable since the clean up step will always remove the entire directory. The second approach gives you more explicit control and makes it easier to move nodes between the “keep” and “discard” lists, but will end up with a lot of node wires in larger graphs. The second approach can also be bundled into an asset more easily, and maybe should be a “Cleanup” TOP node that we ship with Houdini.
Edited by tpetrick - March 18, 2019 16:51:03

Attachments:
fileremove.png (105.4 KB)

User Avatar
Member
24 posts
Joined: May 2015
Offline
Aaah, wonderful ! Since our pipeline implies that we are using various disk drives, using a single folder is out of the question. Attribute-deleting should quite do it. I wasn't that far from the solution in my quest, just couldn't pinpoint that one step.

Thanks !
User Avatar
Member
24 posts
Joined: May 2015
Offline
Just a little note, but AFAIK deleting the “output” attribute doesn't seem to be working as it's the ‘pdg_output’ that will be caried over up to the end, resulting in the MP4 being removed as well. Hitting “delete all attributes” worked for me, though, despite the note in the doc saying the option “does not affect internal attributes such as pdg_index and pdg_output”. Ah well, maybe I'm missing something here, but as long as it works, right ?

I also found out another, maybe simpler way, to get rid of the temp file was simply to play with the tags. Either remove the “file” tag from the MP4 step ( like, leave only video or something entirely different ), or set a different tag for each temp tasks and delete by custom tag at the end.
User Avatar
Member
16 posts
Joined: Dec. 2010
Offline
I am actually solve that, check
https://www.sidefx.com/forum/topic/62404/?page=1#post-277363 [www.sidefx.com]
User Avatar
Member
16 posts
Joined: Dec. 2010
Offline
There example Flush/Remove All files from Render Folder or/and Sim Folder on Begin

So this is nice example, so here on the right side I have nodes pattern file map which is render and sim files, and if Your folder is contain something and this is true its pass and add work item, so its very important to add generic generator to make constant work item, because if Your folder empty and this is false its no any work item generation, and its must stuck and its might stop and don`t move downstream.
About waitforall node…
its very important add waitforall node, because if You execute it again,and this is true its will remove files and add work item and if in plans using wedge(where You set it for example 3), its add extra 1 work item, and its get You 4 items and its going to mess….

Another Example Flush/Remove Files after creation Video if under need

You can find scene there
https://www.sidefx.com/forum/topic/62404/?page=1#post-281012 [www.sidefx.com]

Thanks
Edited by cool1986vfx - April 17, 2019 15:29:20

Attachments:
PDG_Flush_on_Begin.JPG (129.3 KB)
PDG_Remove.JPG (144.8 KB)

  • Quick Links