I try to wrap my head around the following situation. I have a PDG flow that is constantly monitoring a sqlite database and starts can pull between 1 and (a set maximum) of 4 work items at a time. The PDG than spawns 4 processes for another software to process data. So the one startup workitem spawns at 1-4 more workitems.
This works fine for a PDG only environment, but I would like to leverage Deadline for machine/task management. But my head goes into a loop as PDG itself is also a taskmanager. I'm looking for some ideas on how to manage multiple machines, of which some have a certain licensed software while others are meant for other tasks.
Here's a rough sketch of what I think the complexity is.
Tasks spawn work items at the moment. I run a PDG flow it pulls data from a database, then it goes to work. For every task right now I have a PDG flow. But I would want to have a central Dispatcher, that pulls tasks from the database, make the work items and pushes them onto the machines. Some PDG flows spawn commandline tools to manipulate data before running it through another stage. Non of these tasks are trivial basic operations.
I'm looking for some pointers on how to have a single job/task/work item manager to handle all tasks efficiently.
Any info you can give would be of great help. Are there good examples or video's out there that discuss this particular case of PDG use/management?
Thanks for reading!
-Johan