Forums Solaris and Karma XPU shader pre-compile issues

XPU shader pre-compile issues

   722   9   2
User Avatar
brians
Staff
621 posts
Joined: May 2019
Offline
 
Following on from a now-closed H22 beta post regarding XPU shader pre-compile issues:

It has been pointed out that the H22 XPU shader pre-compile feature is running extra slowly than H21. 22.0.398 contains a fix that would have affected multi-GPU users. For anyone affected by the H22 issue with multi-gpu, it would be good to get feedback on 22.0.398 to verify its now running much faster.

But in general (even in H21) its been noted that the shader-precompile feature can run extra slowly on some machines, exhibiting single-thread-like behavior. I'm curious what types of machines this happens on. Its definitly multi-threaded in code, and runs fast on our Linux machines.

Is it possible for anyone to give specific feedback on this?
eg
- is it slow only on Intel? (in which case it might be an e-cores issue. does disabling the e-cores in the bias make it multi-threaded again?) or is it slow on AMD too?
- is it slow only on Windows? Or does Linux also experience this issue?

thanks lots
User Avatar
ronald_a
Member
295 posts
Joined: Aug. 2017
Offline
 
I‘ve put this in an RFE (because of the locked forum), but I‘ll post this here again:

I am on an amd 64core threadripper, two identical gpus, win11. When starting the caching, multiple (i guess 64) karma_cc.exe start up plus the houdini.exe also has cpu load (total cpu load is near 90%). After about a minute, most of the karma.exe are gone with only 1-3 remaining (cpu load is down to single digits). Not sure if the multithreading should be achieved through having multiple karma.exe running or if the processes running in houdini.exe should be more multithreaded.

This makes me wonder if this also affects the compilation time while in interactive mode (for stuff that has not been compiled yet).
User Avatar
brians
Staff
621 posts
Joined: May 2019
Offline
 
ronald_a
When starting the caching, multiple (i guess 64) karma_cc.exe start up plus the houdini.exe

This is strange, it should only be starting 1 karma.exe (and does so on my windows machine)
Do you have some KARMA_XPU_... environment variables set by chance?
User Avatar
ronald_a
Member
295 posts
Joined: Aug. 2017
Offline
 
brians
ronald_a
When starting the caching, multiple (i guess 64) karma_cc.exe start up plus the houdini.exe

This is strange, it should only be starting 1 karma.exe (and does so on my windows machine)
Do you have some KARMA_XPU_... environment variables set by chance?

I do have KARMA_XPU variables set - most notebly KARMA_XPU_THREAD_COMPILE = 1. I disabled all the KARMA_XPU variables for now and now the karma_cc.exes are gone. only one karma.exe remains which runs at a steady 2-4% cpu load.
User Avatar
jsmack
Member
8377 posts
Joined: Sept. 2011
Offline
 
brians
Is it possible for anyone to give specific feedback on this?
eg
- is it slow only on Intel? (in which case it might be an e-cores issue. does disabling the e-cores in the bias make it multi-threaded again?) or is it slow on AMD too?
- is it slow only on Windows? Or does Linux also experience this issue?

I ran precompile on Windows 11 with the following configuration: Intel Core Ultra 7 265k (8P 12E), Nvidia RTX 5080 with 610.88 driver using Houdini 22.0.401
Total time taken was 4 hours 20 minutes 9 seconds. Average core usage was 20%. Usage was almost entirely P-cores and occasionally spiked to all P cores, but was only using 2~4 most of the time. Compiling completed successfully without error.

I don't think the threading is only the issue, it wouldn't be 100x faster if it were using all threads.
User Avatar
ronald_a
Member
295 posts
Joined: Aug. 2017
Offline
 
I deleted any existing cache for H22 and did a fresh run. It took about 7.5 hours (with cpu usage around 2-3%)! Interestingly enough, before doing that, I continued on an existing cache (which was just 7 or eight steps in). After an initial very long step 9, cpu usage increased to about 11% with a remaining time of about 2 hours about 40 steps in. I have seen this behavior before where the cpu usage is somewhat unpredictable. It‘s really weird.
User Avatar
brians
Staff
621 posts
Joined: May 2019
Offline
 
ronald_a
I do have KARMA_XPU variables set - most notebly KARMA_XPU_THREAD_COMPILE = 1. I disabled all the KARMA_XPU variables for now and now the karma_cc.exes are gone. only one karma.exe remains which runs at a steady 2-4% cpu load.

cool
at least the karma_cc.exes are gone

jsmack
I ran precompile on Windows 11 with the following configuration: Intel Core Ultra 7 265k (8P 12E), Nvidia RTX 5080 with 610.88 driver using Houdini 22.0.401
Total time taken was 4 hours 20 minutes 9 seconds. Average core usage was 20%. Usage was almost entirely P-cores and occasionally spiked to all P cores, but was only using 2~4 most of the time.

I'm starting to suspect some kind of file-locking issue within the nvidia cache.

ronald_a
I deleted any existing cache for H22 and did a fresh run. It took about 7.5 hours (with cpu usage around 2-3%)!

Good to know

ronald_a
I have seen this behavior before where the cpu usage is somewhat unpredictable. It‘s really weird.

Yes it is weird :/
Hopefully I'll be getting an e/p-cores machine soon so I can debug more closely. Will report back findings here.

thanks all
User Avatar
Hamilton Meathouse
Member
219 posts
Joined: Nov. 2013
Offline
 
Houdini 22.0.368.
Tried compiling, went through every single precompile lap before telling me it failed. Took 1hr 45min.

CPU utilization sat around 40% (low was 30, high was 60)

CPU - AMD Ryzen 9 7950X 16-Core Processor (32 threads)
GPU - Nvidia RTX3080 Ti, Driver 581.29



[13:59:08] KarmaXPU: Optix Precompile: 548/548 Lap: 1:43:23.66 Left: 0:00
-------- Error Summary --------
PID: 14492
Command: karma --renderer BRAY_HdKarmaXPU -o husk:null_raster -p1 --res 1 1 C:/PROGRA~1/SIDEEF~1/HOUDIN~2.368/houdini/usd/karma/kma_precompile_shaders.usda
-------- 548 total errors --------
[12:15:47] KarmaXPU: Optix Precompile: 1/548 Lap: 0:02.56 Left: 23:19.77
[12:15:49] KarmaXPU: Optix Precompile: 2/548 Lap: 0:03.78 Left: 17:12.72
[12:15:56] KarmaXPU: Optix Precompile: 3/548 Lap: 0:11.25 Left: 34:03.40


EDIT: Looks like it worked however?


Utilization was high for cores 0-15, lower for 16-31:
Edited by Hamilton Meathouse - yesterday 17:41:54

Attachments:
optixCache.png (46.2 KB)
houdini_xpu_compile_01.jpg (142.6 KB)

User Avatar
jason_iversen
Member
13287 posts
Joined: July 2005
Offline
 


Houdini FX Version 22.0.408

Open Source Information:  https://www.sidefx.com/docs/houdini/licenses/index.html

Operating System:         Microsoft Windows 11 Pro x64
Processor:                11th Gen Intel(R) Core(TM) i7-11850H @ 2.50GHz
Number of Cores:          16
Physical Memory:          63.71 GB
Number of Screens:        2
    Screen 0:             3840 x 2160 at 0,0
    Work Area 0:          3840 x 2088 at 0,0
    Screen 0 DPI:         163.3
    Screen 1:             3840 x 2400 at 3840,0
    Work Area 1:          3840 x 2280 at 3840,0
    Screen 1 DPI:         283.5

Platform Build:           windows-x86_64-cl19.42
Python Version:           3.13
Qt Version:               6.8.3
USD Version:              26.05
USD git URL:              https://github.com/sideeffects/USD.git
USD git Revision:         ee7b08cf5dae8ff0a206ee907be0326288ab59bf
UI Scale:                 1.500

Detected GPU:             NVIDIA Consumer
                          8192 MB
                          610.47.0.0


Vulkan Vendor:            NVIDIA Corporation
Vulkan Device:            NVIDIA GeForce RTX 3070 Laptop GPU
Vulkan Version:           1.4.341
Vulkan Loader Version:    1.3.0
Device Heaps:             8018 MB

Attachments:
Precompile.png (687.9 KB)

Jason Iversen, Technology Supervisor & R&D Lead @ WetaFX
User Avatar
Mirko Jankovic
Member
315 posts
Joined: Aug. 2015
Online
 
ugh.. H22.0.409 Daily build

edit: it did drop down significantly after a while:

KarmaXPU: Optix Precompile: 436/446 Lap: 1:13:41.27 Left: 1:41.41
KarmaXPU: Optix Precompile: 437/446 Lap: 1:13:49.75 Left: 1:31.23
KarmaXPU: Optix Precompile: 438/446 Lap: 1:13:56.18 Left: 1:21.03
Edited by Mirko Jankovic - today 12:17:11

Attachments:
Screenshot_20260812_173444-2.png (1.2 MB)

  • Quick Links