After a lot of trouble shooting I realized that any process that takes less than 60 seconds doesn't repeat and any process that takes longer than 60 seconds does repeat. The longer the process, the worst this. Let's say a process takes ten minutes, then that will cause the same file to be re-written 10 times at minimum.
I replicated this in the attached HIP file, where there is a simple LOP context workflow that creates and modified some geometry and then has a python LOP that injects a certain number of seconds delay to the processing time. One path injects 55 seconds, and the other path injects 65 seconds.
And then in TOPS context I activate a Service with a pool size of 24 and use it with a ROP USD Output node to cook out the geometry to usd files that are time stamped down to the millisecond so that each file gets a unique filename even if it is the result of a redundant generation.
If I process the ROP USD Output that points to the 55 second delay I get 100 files, if I point it to the one with the 65 second delay I get 360 output files. If I inject a 5 minute delay it just generates thousands because the system gets gummed up and each job gets submitted dozens of times.
In all cases the only difference between the generations is the length of the delay, which means that I can't use services to process anything but the smallest operations with the shortest processing time. Any heavy duty processing quickly becomes exponentially long in terms of processing time.
_____________________________________________________
The .hip file and the above description of the problem is the hard evidence created and written by me (a human), but I also have this AI slop summary of a long conversation I had with ChatGPT where I figured all this out that goes in to a lot more detail (perhaps too much detail), but I'm including it here in case any part of it is useful in figuring out the root cause of the problem.
## PDG Services appear to re-submit long-running work items while they are still executing
I have been tracking a problem where PDG Services repeatedly execute the same work item instead of progressing normally through the graph.
The visible symptom is that an output file is written successfully, then immediately rewritten several times. This initially looked like an output-path problem, stale parameter evaluation, caching issue, or several work items accidentally resolving to the same filename.
That does not appear to be what is happening.
### What appears to be happening
PDG Services use persistent Houdini processes connected through Houdini’s message-queue system. PDG sends a work request to one of those processes, the process performs the operation, and a reply eventually reports that the work has finished.
For long-running operations, the original request appears to remain unanswered for the duration of the cook. While that first execution is still running, the messaging layer sends additional copies of the request at roughly one-minute intervals.
Those copies are not being recognized as retransmissions of work that is already running. Instead, each copy is placed in the service client’s queue as another executable request.
The resulting sequence appears to be:
1. PDG sends a work item to a service client.
2. The service client begins executing it.
3. The execution takes longer than the message retry interval.
4. The still-unanswered request is transmitted again.
5. More copies accumulate while the original execution continues.
6. The original execution finishes successfully.
7. The service client then processes the queued copies.
8. Each copy executes the same work item again and writes the same output.
This explains the strange behavior where the output seems stuck on one file for an extended period and then eventually starts progressing again. The service is spending that time clearing duplicate executions that accumulated while the first cook was running.
### Why the one-minute interval matters
Houdini’s published interfaces indicate that PDG Services use an NNG-based polling client.
NNG’s request/reply protocol can automatically retransmit a request when it has not received a reply. Duplicate delivery is an expected possibility in that type of transport, which means the operation receiving the request must be idempotent or must deduplicate repeated request IDs.
The almost exact one-minute spacing in the Houdini MQ logs strongly matches that behavior.
The public logs do not include enough information to prove whether the retransmission is coming directly from NNG’s built-in resend behavior or from a SideFX retry loop built on top of it. However, they clearly show the important part: repeated messages are being delivered while the first execution is outstanding, and the service client subsequently executes the same PDG work item once for every delivered copy.
### How I ruled out output-path problems
I first replaced the original output expressions with fully assembled PDG string attributes so that every work item already contained its final absolute path.
I then added logging immediately before the USD output executed. The log recorded:
* The current PDG work-item ID
* The assembled output-path attribute
* The value evaluated by the USD output parameter
* The service process handling the work
The correct work-item-specific path was present immediately before every export.
More importantly, the logs showed the same work-item ID and the same correctly resolved path being executed repeatedly. Different work-item IDs were not colliding on one path. One work item was being invoked multiple times.
I also checked the combined client logs across a large number of work items. Each work-item ID had one unique output path, and each output path belonged to one work-item ID. Despite that one-to-one mapping, the individual items were still executed multiple times.
Adding timestamps to the filenames made the repeated executions visible as separate completed files instead of allowing them to overwrite one another.
### Other causes that were tested
The problem was reproduced after removing or simplifying the surrounding network, so it was not caused by the production material-assignment script or the complexity of the Solaris stage.
The following were also eliminated:
* Frame batching
* Multiple frames per work item
* Block Begin nodes
* Dynamic filename evaluation
* Output-path collisions
* Network-drive safe saving
* Upstream-file cache invalidation
* The `--native` service option
* The particular USD geometry being exported
* The particular type of PDG service
* A normal scheduler retry after failure
The same operation works correctly when run out of process. The repeated execution appears specifically when it is dispatched through PDG Services.
A minimal reproduction can replace the expensive production operation with an intentional delay followed by a USD output. This demonstrates that the important trigger is the duration of the service request, not anything special about the geometry or material network.
The service logs refer to the operation as `hou.Rop.render`, but that is simply Houdini’s generic method for executing a ROP. In this case, the ROP is exporting USD geometry; it is not performing an image render.
### What the MQ logging showed
After enabling full service-client and MQ logging, most of the MQ log consisted of repetitive polling messages. Filtering that noise revealed a very consistent pattern:
* A work request was delivered to a particular service client.
* The service client began processing it.
* Additional same-signature messages were delivered to that client at approximately one-minute intervals.
* Once the original execution finished, the client executed the same work item repeatedly.
* The number of repeated client executions corresponded to the number of messages delivered while the original execution was running.
The logging also suggests that these unanswered duplicate messages continue accumulating. Eventually, the number of deliveries without corresponding replies approaches the MQ server’s configured capacity. At that point, client activity stops, followed later by the MQ server automatically shutting down because it no longer sees network activity.
The logs do not explicitly identify connection exhaustion as the shutdown cause, so that final part needs confirmation from SideFX. The relationship between the accumulated unanswered requests and the configured limit is nevertheless extremely suspicious.
### Why the available settings do not fix it
The Service Connection Timeout controls how long a service client is allowed to take while starting and connecting. It is not a maximum work-item duration and does not control the apparent one-minute retransmission.
ROP Fetch caching only determines whether an item should be scheduled initially. It cannot prevent execution of duplicate requests that have already been transmitted and queued.
Scheduler retry settings apply after a task reports failure. In this case, the copies are transmitted while the original task is still running successfully.
The documented service arguments alter how work-item data and wedge attributes are loaded. They do not expose message acknowledgement, retransmission, or deduplication controls.
Pre-render and post-render scripts also operate too late in the process. By the time those scripts run, the service request has already been delivered. They cannot acknowledge the original transport request or remove copies already waiting in the queue.
A custom lock file or completion marker could prevent some duplicate output writes, but it would only conceal the most visible symptom. The duplicate messages would still accumulate, the service would still waste time processing them, and the MQ system could still eventually stall. Locking would also introduce its own problems involving crashes, partial outputs, stale locks, and intentional retries.
### What probably needs to change in Houdini
The service protocol needs to separate acceptance of a request from completion of the operation.
When a service client receives a work item, it should promptly acknowledge that the request has been accepted. The actual cook can then continue asynchronously and report completion through a separate status or result message.
Every logical PDG execution attempt should also have a stable identifier. If the transport retransmits the request, the copy must retain the same identifier. The broker and service client can then recognize that the work is already queued, running, or completed instead of executing it again.
An intentional PDG retry after a real failure should receive a new attempt identifier. That would distinguish a legitimate retry from a transport-level replay.
SideFX could also disable automatic retransmission for these non-idempotent execution requests, but disabling retransmission by itself would not be a complete solution. A proper implementation should combine:
* Immediate acknowledgement of request acceptance
* Stable execution-attempt identifiers
* Deduplication of queued, running, and completed requests
* Explicit PDG-controlled retries
* Logging that distinguishes original deliveries from retransmissions
There does not appear to be a supported parameter, environment variable, Python API, service argument, or MQ command-line option that exposes the relevant transport behavior. Unless SideFX identifies an undocumented control, correcting the underlying signaling requires a Houdini update.
The essential issue is not that PDG Services cannot theoretically run operations longer than one minute. It is that the affected service implementation appears to leave the execution request unacknowledged during a long cook while also allowing transport retransmissions to become additional executions.
A transport retry must never create another ROP invocation for the same logical PDG attempt.