Following on from a now-closed H22 beta post regarding XPU shader pre-compile issues:

It has been pointed out that the H22 XPU shader pre-compile feature is running extra slowly than H21. 22.0.398 contains a fix that would have affected multi-GPU users. For anyone affected by the H22 issue with multi-gpu, it would be good to get feedback on 22.0.398 to verify its now running much faster.

But in general (even in H21) its been noted that the shader-precompile feature can run extra slowly on some machines, exhibiting single-thread-like behavior. I'm curious what types of machines this happens on. Its definitly multi-threaded in code, and runs fast on our Linux machines.

Is it possible for anyone to give specific feedback on this?
eg
- is it slow only on Intel? (in which case it might be an e-cores issue. does disabling the e-cores in the bias make it multi-threaded again?) or is it slow on AMD too?
- is it slow only on Windows? Or does Linux also experience this issue?

thanks lots

I‘ve put this in an RFE (because of the locked forum), but I‘ll post this here again:

I am on an amd 64core threadripper, two identical gpus, win11. When starting the caching, multiple (i guess 64) karma_cc.exe start up plus the houdini.exe also has cpu load (total cpu load is near 90%). After about a minute, most of the karma.exe are gone with only 1-3 remaining (cpu load is down to single digits). Not sure if the multithreading should be achieved through having multiple karma.exe running or if the processes running in houdini.exe should be more multithreaded.

This makes me wonder if this also affects the compilation time while in interactive mode (for stuff that has not been compiled yet).

ronald_a
When starting the caching, multiple (i guess 64) karma_cc.exe start up plus the houdini.exe

This is strange, it should only be starting 1 karma.exe (and does so on my windows machine)
Do you have some KARMA_XPU_... environment variables set by chance?

brians

ronald_a
When starting the caching, multiple (i guess 64) karma_cc.exe start up plus the houdini.exe

This is strange, it should only be starting 1 karma.exe (and does so on my windows machine)
Do you have some KARMA_XPU_... environment variables set by chance?

I do have KARMA_XPU variables set - most notebly KARMA_XPU_THREAD_COMPILE = 1. I disabled all the KARMA_XPU variables for now and now the karma_cc.exes are gone. only one karma.exe remains which runs at a steady 2-4% cpu load.

brians
Is it possible for anyone to give specific feedback on this?
eg
- is it slow only on Intel? (in which case it might be an e-cores issue. does disabling the e-cores in the bias make it multi-threaded again?) or is it slow on AMD too?
- is it slow only on Windows? Or does Linux also experience this issue?

I ran precompile on Windows 11 with the following configuration: Intel Core Ultra 7 265k (8P 12E), Nvidia RTX 5080 with 610.88 driver using Houdini 22.0.401
Total time taken was 4 hours 20 minutes 9 seconds. Average core usage was 20%. Usage was almost entirely P-cores and occasionally spiked to all P cores, but was only using 2~4 most of the time. Compiling completed successfully without error.

I don't think the threading is only the issue, it wouldn't be 100x faster if it were using all threads.

I deleted any existing cache for H22 and did a fresh run. It took about 7.5 hours (with cpu usage around 2-3%)! Interestingly enough, before doing that, I continued on an existing cache (which was just 7 or eight steps in). After an initial very long step 9, cpu usage increased to about 11% with a remaining time of about 2 hours about 40 steps in. I have seen this behavior before where the cpu usage is somewhat unpredictable. It‘s really weird.

ronald_a
I do have KARMA_XPU variables set - most notebly KARMA_XPU_THREAD_COMPILE = 1. I disabled all the KARMA_XPU variables for now and now the karma_cc.exes are gone. only one karma.exe remains which runs at a steady 2-4% cpu load.

cool
at least the karma_cc.exes are gone

jsmack
I ran precompile on Windows 11 with the following configuration: Intel Core Ultra 7 265k (8P 12E), Nvidia RTX 5080 with 610.88 driver using Houdini 22.0.401
Total time taken was 4 hours 20 minutes 9 seconds. Average core usage was 20%. Usage was almost entirely P-cores and occasionally spiked to all P cores, but was only using 2~4 most of the time.

I'm starting to suspect some kind of file-locking issue within the nvidia cache.

ronald_a
I deleted any existing cache for H22 and did a fresh run. It took about 7.5 hours (with cpu usage around 2-3%)!

Good to know

ronald_a
I have seen this behavior before where the cpu usage is somewhat unpredictable. It‘s really weird.

Yes it is weird :/
Hopefully I'll be getting an e/p-cores machine soon so I can debug more closely. Will report back findings here.

thanks all

Houdini 22.0.368.
Tried compiling, went through every single precompile lap before telling me it failed. Took 1hr 45min.

CPU utilization sat around 40% (low was 30, high was 60)

CPU - AMD Ryzen 9 7950X 16-Core Processor (32 threads)
GPU - Nvidia RTX3080 Ti, Driver 581.29





EDIT: Looks like it worked however?


Utilization was high for cores 0-15, lower for 16-31:

ugh.. H22.0.409 Daily build

edit: it did drop down significantly after a while:

KarmaXPU: Optix Precompile: 436/446 Lap: 1:13:41.27 Left: 1:41.41
KarmaXPU: Optix Precompile: 437/446 Lap: 1:13:49.75 Left: 1:31.23
KarmaXPU: Optix Precompile: 438/446 Lap: 1:13:56.18 Left: 1:21.03

Hamilton Meathouse
Houdini 22.0.368.
Tried compiling, went through every single precompile lap before telling me it failed. Took 1hr 45min.

22.0.398 has this fixed

Hi guys

After much debugging (and a new PC ) we've now fixed this issue. It was a fault in the NVidia CUDA/Optix compiler affecting Blackwell GPUs.

22.0.422 will have the fix. Please give it a try and let us know here if your precompile times are back to normal.

Thanks very much for your patience on this one.

I'm probably not the target audience with my RTX 3070 Laptop card, but:

I'm afraid, this still does not work for me. I'm also wondering about the fact that it had something to do with blackwell as I don't have blackwell gpus in that particular machine.

jason_iversen
I'm probably not the target audience with my RTX 3070 Laptop card, but:

At least we have 91% utilization of your CPU now. I'll try to bring the cost down further over the coming days/weeks.
some questions:
- is this a regression for you Jason? Or has it always taken this long? (ie even in H21?)
- whats the temperature like on your CPU while this is running? (something like HWinfo can report this). Is it possible the Laptop is old and the CPU is overheating/throttling?

ronald_a
I'm afraid, this still does not work for me.

What are the symptoms now? Are you able to post a screenshot like what Jason did?
Also, can you confirm you no longer have any KARMA_XPU_... environment variables set?

thanks

brians

jason_iversen
I'm probably not the target audience with my RTX 3070 Laptop card, but:

At least we have 91% utilization of your CPU now. I'll try to bring the cost down further over the coming days/weeks.
some questions:
- is this a regression for you Jason? Or has it always taken this long? (ie even in H21?)
- whats the temperature like on your CPU while this is running? (something like HWinfo can report this). Is it possible the Laptop is old and the CPU is overheating/throttling?

ronald_a
I'm afraid, this still does not work for me.

What are the symptoms now? Are you able to post a screenshot like what Jason did?
Also, can you confirm you no longer have any KARMA_XPU_... environment variables set?

thanks

Well the symptoms are just like before - only a fraction of the available CPU cores are used (about 20/64cores (128threads)). Also, the used cores do not seem to be maxed out but work at around 10-50%. I can confirm that no KARMA_XPU variables are used. If I can help by running some debug code here, I am more than happy to do so.

I've got full utilization here.
ryzen 5950x 16cores/32threads, 128G ram, dual 4090
nobara linux
All done in
KarmaXPU: Optix Precompile: 446/446 Lap: 6:16.20 Left: 0:00

It's about the same for me. It got about an hour faster, but still takes an order of magnitude longer than anyone else on here-about two hours. Utilization varies between 40 and 80% now instead of 10-35% though. Scheduling is bad, E-cores are getting maxed out before P-cores. Most E-cores sit idle though with all but 2 P-cores sitting at 45-60% non-idle power state. Total package power usage is only around 60-80 watts, less than a third full power. This CPU uses around 200 while rendering.

brians
- is this a regression for you Jason? Or has it always taken this long? (ie even in H21?)

Yes, sadly, according to this post above [www.sidefx.com], I was getting ~3 hrs predicted before .422. Now it's over 20 hrs...

Ah wait -- I updated my display driver, made sure I wasn't in battery-save mode and now it's back in the slot:

jason_iversen
Ah wait -- I updated my display driver, made sure I wasn't in battery-save mode and now it's back in the slot:

OMG...
I'm glad its now working again, thanks for testing Jason.

ronald_a
Well the symptoms are just like before - only a fraction of the available CPU cores are used (about 20/64cores (128threads)). Also, the used cores do not seem to be maxed out but work at around 10-50%.

That just leaves ronald_a now.
I think we can try a few things
eg
- do you have some kind of power-save mode enabled on your computer?
- what if you do a KarmaXPU render using only the CPU (ie KARMA_XPU_DEVICES=embree), do all the cores light up then?
- what happens if you update to latest driver as well? (616.56 dropped on Wednesday)
- what happens if you set KARMA_XPU_THREAD_COMPILE=1 again? do you see all cores light up initially as you did originally? (it means it seems to parallelize when we fire up multiple processes, rather than multiple threads).
- what total time do you actually get? Perhaps task-manager is saying low-usage, but in reality its working fine?