I'm in Solaris and using "Render all frames in a single pass" because it's much faster. It starts out by making a USD cache which can take a while. For a longer frame range like 4000 When the actual rending is about to start it will crash saying:
KarmaXPU: Failed to initialize CUDA
Error creating optix denoiser
Unable to create denoise image filter plugin

But then I'll give it a frame range of 1000 and this doesn't happen. I do have most of my geo that comes into Solaris cached. My CPU ram (and system ram) remains very low throughout all of it. I tried using a different version of Houdini and different drivers. Any ideas?

So just to repeat, you're saying that CUDA/Optix work in the viewport, and with low frame-ranges (eg 1000)
But you get this "Failed to initialize CUDA" error if you render over 4000 frames?

Are you on linux or windows?
Are you able to see what kind of memory usage you're getting on the GPU during this caching process?

That's correct. I'm on windows. System and GPU memory is very low during this.
I bring a few things into Solaris with Sop import and cached 9 out of 10 of them. I just removed the one that isnt USD cached and has some geo processing happing in SOP each frame. With it removed and only USD caches being used the problem goes away. Still very strange. I've had everything coming in from sops before and the rendering issue didnt happen then.

This is strange, I'm guessing something to do with the OpenCL at the earlier stages is putting the driver in a bad state, which later affects XPU.

Can you please post the exact CUDA error string you get?
And better yet, post a screenshot of it (this forum often strips out the error message for some reason, something to do with the square brackets)

Also, try forcing OpenCL to use the CPU with the following envvar.
HOUDINI_OCL_DEVICETYPE=CPU

Thanks