Following on from a now-closed H22 beta post regarding XPU shader pre-compile issues:
It has been pointed out that the H22 XPU shader pre-compile feature is running extra slowly than H21. 22.0.398 contains a fix that would have affected multi-GPU users. For anyone affected by the H22 issue with multi-gpu, it would be good to get feedback on 22.0.398 to verify its now running much faster.
But in general (even in H21) its been noted that the shader-precompile feature can run extra slowly on some machines, exhibiting single-thread-like behavior. I'm curious what types of machines this happens on. Its definitly multi-threaded in code, and runs fast on our Linux machines.
Is it possible for anyone to give specific feedback on this?
eg
- is it slow only on Intel? (in which case it might be an e-cores issue. does disabling the e-cores in the bias make it multi-threaded again?) or is it slow on AMD too?
- is it slow only on Windows? Or does Linux also experience this issue?
thanks lots
Full Version: XPU shader pre-compile issues
I‘ve put this in an RFE (because of the locked forum), but I‘ll post this here again:
I am on an amd 64core threadripper, two identical gpus, win11. When starting the caching, multiple (i guess 64) karma_cc.exe start up plus the houdini.exe also has cpu load (total cpu load is near 90%). After about a minute, most of the karma.exe are gone with only 1-3 remaining (cpu load is down to single digits). Not sure if the multithreading should be achieved through having multiple karma.exe running or if the processes running in houdini.exe should be more multithreaded.
This makes me wonder if this also affects the compilation time while in interactive mode (for stuff that has not been compiled yet).
I am on an amd 64core threadripper, two identical gpus, win11. When starting the caching, multiple (i guess 64) karma_cc.exe start up plus the houdini.exe also has cpu load (total cpu load is near 90%). After about a minute, most of the karma.exe are gone with only 1-3 remaining (cpu load is down to single digits). Not sure if the multithreading should be achieved through having multiple karma.exe running or if the processes running in houdini.exe should be more multithreaded.
This makes me wonder if this also affects the compilation time while in interactive mode (for stuff that has not been compiled yet).
ronald_a
When starting the caching, multiple (i guess 64) karma_cc.exe start up plus the houdini.exe
This is strange, it should only be starting 1 karma.exe (and does so on my windows machine)
Do you have some KARMA_XPU_... environment variables set by chance?
briansronald_a
When starting the caching, multiple (i guess 64) karma_cc.exe start up plus the houdini.exe
This is strange, it should only be starting 1 karma.exe (and does so on my windows machine)
Do you have some KARMA_XPU_... environment variables set by chance?
I do have KARMA_XPU variables set - most notebly KARMA_XPU_THREAD_COMPILE = 1. I disabled all the KARMA_XPU variables for now and now the karma_cc.exes are gone. only one karma.exe remains which runs at a steady 2-4% cpu load.
brians
Is it possible for anyone to give specific feedback on this?
eg
- is it slow only on Intel? (in which case it might be an e-cores issue. does disabling the e-cores in the bias make it multi-threaded again?) or is it slow on AMD too?
- is it slow only on Windows? Or does Linux also experience this issue?
I ran precompile on Windows 11 with the following configuration: Intel Core Ultra 7 265k (8P 12E), Nvidia RTX 5080 with 610.88 driver using Houdini 22.0.401
Total time taken was 4 hours 20 minutes 9 seconds. Average core usage was 20%. Usage was almost entirely P-cores and occasionally spiked to all P cores, but was only using 2~4 most of the time. Compiling completed successfully without error.
I don't think the threading is only the issue, it wouldn't be 100x faster if it were using all threads.
I deleted any existing cache for H22 and did a fresh run. It took about 7.5 hours (with cpu usage around 2-3%)! Interestingly enough, before doing that, I continued on an existing cache (which was just 7 or eight steps in). After an initial very long step 9, cpu usage increased to about 11% with a remaining time of about 2 hours about 40 steps in. I have seen this behavior before where the cpu usage is somewhat unpredictable. It‘s really weird.
ronald_a
I do have KARMA_XPU variables set - most notebly KARMA_XPU_THREAD_COMPILE = 1. I disabled all the KARMA_XPU variables for now and now the karma_cc.exes are gone. only one karma.exe remains which runs at a steady 2-4% cpu load.
cool
at least the karma_cc.exes are gone
jsmack
I ran precompile on Windows 11 with the following configuration: Intel Core Ultra 7 265k (8P 12E), Nvidia RTX 5080 with 610.88 driver using Houdini 22.0.401
Total time taken was 4 hours 20 minutes 9 seconds. Average core usage was 20%. Usage was almost entirely P-cores and occasionally spiked to all P cores, but was only using 2~4 most of the time.
I'm starting to suspect some kind of file-locking issue within the nvidia cache.
ronald_a
I deleted any existing cache for H22 and did a fresh run. It took about 7.5 hours (with cpu usage around 2-3%)!
Good to know
ronald_a
I have seen this behavior before where the cpu usage is somewhat unpredictable. It‘s really weird.
Yes it is weird :/
Hopefully I'll be getting an e/p-cores machine soon so I can debug more closely. Will report back findings here.
thanks all
Houdini 22.0.368.
Tried compiling, went through every single precompile lap before telling me it failed. Took 1hr 45min.
CPU utilization sat around 40% (low was 30, high was 60)
CPU - AMD Ryzen 9 7950X 16-Core Processor (32 threads)
GPU - Nvidia RTX3080 Ti, Driver 581.29
EDIT: Looks like it worked however?
Utilization was high for cores 0-15, lower for 16-31:
Tried compiling, went through every single precompile lap before telling me it failed. Took 1hr 45min.
CPU utilization sat around 40% (low was 30, high was 60)
CPU - AMD Ryzen 9 7950X 16-Core Processor (32 threads)
GPU - Nvidia RTX3080 Ti, Driver 581.29
[13:59:08] KarmaXPU: Optix Precompile: 548/548 Lap: 1:43:23.66 Left: 0:00 -------- Error Summary -------- PID: 14492 Command: karma --renderer BRAY_HdKarmaXPU -o husk:null_raster -p1 --res 1 1 C:/PROGRA~1/SIDEEF~1/HOUDIN~2.368/houdini/usd/karma/kma_precompile_shaders.usda -------- 548 total errors -------- [12:15:47] KarmaXPU: Optix Precompile: 1/548 Lap: 0:02.56 Left: 23:19.77 [12:15:49] KarmaXPU: Optix Precompile: 2/548 Lap: 0:03.78 Left: 17:12.72 [12:15:56] KarmaXPU: Optix Precompile: 3/548 Lap: 0:11.25 Left: 34:03.40
EDIT: Looks like it worked however?
Utilization was high for cores 0-15, lower for 16-31:
Houdini FX Version 22.0.408
Open Source Information: https://www.sidefx.com/docs/houdini/licenses/index.html
Operating System: Microsoft Windows 11 Pro x64
Processor: 11th Gen Intel(R) Core(TM) i7-11850H @ 2.50GHz
Number of Cores: 16
Physical Memory: 63.71 GB
Number of Screens: 2
Screen 0: 3840 x 2160 at 0,0
Work Area 0: 3840 x 2088 at 0,0
Screen 0 DPI: 163.3
Screen 1: 3840 x 2400 at 3840,0
Work Area 1: 3840 x 2280 at 3840,0
Screen 1 DPI: 283.5
Platform Build: windows-x86_64-cl19.42
Python Version: 3.13
Qt Version: 6.8.3
USD Version: 26.05
USD git URL: https://github.com/sideeffects/USD.git
USD git Revision: ee7b08cf5dae8ff0a206ee907be0326288ab59bf
UI Scale: 1.500
Detected GPU: NVIDIA Consumer
8192 MB
610.47.0.0
Vulkan Vendor: NVIDIA Corporation
Vulkan Device: NVIDIA GeForce RTX 3070 Laptop GPU
Vulkan Version: 1.4.341
Vulkan Loader Version: 1.3.0
Device Heaps: 8018 MB
ugh.. H22.0.409 Daily build
edit: it did drop down significantly after a while:
KarmaXPU: Optix Precompile: 436/446 Lap: 1:13:41.27 Left: 1:41.41
KarmaXPU: Optix Precompile: 437/446 Lap: 1:13:49.75 Left: 1:31.23
KarmaXPU: Optix Precompile: 438/446 Lap: 1:13:56.18 Left: 1:21.03
edit: it did drop down significantly after a while:
KarmaXPU: Optix Precompile: 436/446 Lap: 1:13:41.27 Left: 1:41.41
KarmaXPU: Optix Precompile: 437/446 Lap: 1:13:49.75 Left: 1:31.23
KarmaXPU: Optix Precompile: 438/446 Lap: 1:13:56.18 Left: 1:21.03
Hamilton Meathouse
Houdini 22.0.368.
Tried compiling, went through every single precompile lap before telling me it failed. Took 1hr 45min.
22.0.398 has this fixed