To make it simple, i getting weird behavior from Houdini with rendering huge amount of texture files.

What i have:
one big monster, with 70 UDIMs texture in TIF and EXR.

What is the problem:
- rendering the monster with 2k textures (4gb total filesize) works fine
- rendering the monster with 4k textures (9gb total filesize) doesnt work, it loads the textures inside RAM memory and then it hangs and doesnt render (left it over night)
- rendering the monster with 4k textures converted to RAT works - but unusable in lookdev process due changing textures on the fly


the same exist when rendering using Arnold, once i load the 2k udim textures renders fine, with 4k textures just hangs

I did test renders with the same textures in Softimage + Arnold and no problem at all.


Anyone else seen bad behavior with huge amount of textures?

Using H14 14.0.258

BTW its not the machine fault, i do have enough free RAM and enough of CPU power

This is probably because the rat files are mip mapped. I am not sure if mip mapped exr files help yet, this is a post I remember on that:

https://www.sidefx.com/index.php?option=com_forum&Itemid=172&page=viewtopic&t=35557&sid=00cca6ba5b8de30c80258bba7ebf047d [sidefx.com]

For arnold, try to make sure you are using mip mapped exr's.

Hope that points in the right direction.

Cheers,
koen

If you are using a farm to render this you should be using mip mapped .rat files it will be more efficient.
I am interested as to why your using 70 udim with 4k textures. Surely the whole point of udim is to use variation in your texture size to get detail where its needed and use less where not.

- rendering the monster with 4k textures converted to RAT works - but unusable in lookdev process due changing textures on the fly

You can solve the conversion fiddle in a shell by just writing a push button script.

r

thank you guys for reply.
sure for renderfarm i will convert to RAT, but then im working localy on lookdev i want to use the native output format (TIFF, EXR) and not the RAT, that i can open only in Houdini.

I do have 70 4k textures because i have huge giant, that is going to be used for distant shot as well as for extreme closeup to different parts of body.

Ahhh , cool. Well I would look at push button scripting a lower res set of textures for those far off shots, it will make your farm throughput much faster.

Rob

martinkindl83
the same exist when rendering using Arnold, once i load the 2k udim textures renders fine, with 4k textures just hangs

I did test renders with the same textures in Softimage + Arnold and no problem at all.

There is currently a bug in HtoA that prevents the updating of the textures in IPR as in SItoA (ticket #469), this should be sorted in the next 1.4.1 release. Other than that, it should really be identical behavior in both packages.

I'd be really keen to get a repro scene, are in based in London by any chance?

Also, if you can get your texture editor to save directly to a tiled and mip-mapped EXR or TIFF, Arnold will pick that up and you won't be using as much memory.

after two days spend of testing it seems that its Houdinis fault not renderer. It seems that its loading textures extremely slow (small chunks maybe?) from network.
Softimage loads all the textures (90% network utilization) and then starts rendering (100% CPU). Houdini (Arnold/Mantra) is loading textures at 5% of network capacity and renders (3-5% CPU utilization) during the loading process.

The time difference is huge.
Loading textures from local hdd and rendering locally gives me 7minutes per frame. Loading the textures from network (still rendering locally) - 3hours per frame (network utilized 3-20% on 1Gb link).

Tested on H14 258 and H14 327. with RAT files. Result is the same.

I cant provide the scene, but its generic fault, just take 4x 70 textures, plug them into shader and apply it on any model with UDIM.

And just saying again, on Softimage there is no issue even loading 3x100 8k UDIM textures (previous project)

Ok, I'll try to repro with Arnold, but it makes no sense to get different render times or CPU utilization between rendering to Arnold with Softimage or Houdini.

martinkindl83
after two days spend of testing it seems that its Houdinis fault not renderer. It seems that its loading textures extremely slow (small chunks maybe?) from network.
Softimage loads all the textures (90% network utilization) and then starts rendering (100% CPU). Houdini (Arnold/Mantra) is loading textures at 5% of network capacity and renders (3-5% CPU utilization) during the loading process.

The time difference is huge.
Loading textures from local hdd and rendering locally gives me 7minutes per frame. Loading the textures from network (still rendering locally) - 3hours per frame (network utilized 3-20% on 1Gb link).

Tested on H14 258 and H14 327. with RAT files. Result is the same.

I cant provide the scene, but its generic fault, just take 4x 70 textures, plug them into shader and apply it on any model with UDIM.

And just saying again, on Softimage there is no issue even loading 3x100 8k UDIM textures (previous project)

I cannot reproduce this here. I used the NASA images of the earth (an 8Kx4K version). I can supply the image if you need. Making copies using the python script
import sys, shutil, os

SRC = ‘world-july-8K.rat’
if not os.path.exists(SRC):
os.system('icp world-july-8K.jpg %s' % SRC)

for i in range(100):
dest = ‘map-%04d.rat’ % i
print i
if not os.path.exists(dest):
shutil.copy(SRC, dest)


Then, using the attached .hip file, my render times were:
Local Textures:

reading geometry from stdin
/obj/geo1/switch1 (2401 primitives)

Generating Image: ip (1280x720)
Plane: Cf+Af (16-bit float)
SampleFilter: alpha
PixelFilter: gaussian -w 2
VEX Type: vector4
Gamma: 1
Load Time: 1.16u 0.25s 0.62r
Memory: 260.21 MB
Ray Counts:
8,330,436 primary rays 9.04 rays/pixel
8,209,547 occlusion rays 8.91 rays/pixel
8,003,977 shading rays 8.68 rays/pixel
24,543,960 total rays 26.63 rays/pixel
Unified Cache: 13.82 MB of 7.84 GB used
In Cache Faults Generated Item Size
TBF Texture: 13.82 MB 2141 13.82 MB 6.61 KB
Render Time: 1:14.99u 0.91s 8.31r
Memory: 1.17 GB

Peak Geometry Objects: 2


Network Textures

reading geometry from stdin
/obj/geo1/switch1 (2401 primitives)

Generating Image: ip (1280x720)
Plane: Cf+Af (16-bit float)
SampleFilter: alpha
PixelFilter: gaussian -w 2
VEX Type: vector4
Gamma: 1
Load Time: 1.18u 0.18s 0.58r
Memory: 260.21 MB
Ray Counts:
8,330,436 primary rays 9.04 rays/pixel
8,209,560 occlusion rays 8.91 rays/pixel
8,003,680 shading rays 8.68 rays/pixel
24,543,676 total rays 26.63 rays/pixel
Unified Cache: 13.88 MB of 7.84 GB used
In Cache Faults Generated Item Size
TBF Texture: 13.88 MB 2150 13.88 MB 6.61 KB
Render Time: 1:14.36u 1.12s 8.33r
Memory: 1.17 GB

Peak Geometry Objects: 2


If you are using non .rat files, mantra will load the images and convert them to internal .rat files for proper filtering. This will use considerably more memory and impact performance severely if you have a lot of large textures.

Please make sure you're using .rat files as textures using (iinfo on the image file). Mantra should also warn you that its converting textures if you set the verbosity of the render up a little bit.

I will be able to check your file in 4 hours. But from description it looks like you are loading just one texture?
I'm loading 210 different 4k images in Rat.

I'm well aware about the internal rat conversion all the issues associated with it.

martinkindl83
I will be able to check your file in 4 hours. But from description it looks like you are loading just one texture?
I'm loading 210 different 4k images in Rat.

I'm well aware about the internal rat conversion all the issues associated with it.

100 different 8K .rat files. The same image, but separate disk files (5.2 GB of disk space).

They could be different files, but I don't happen to have 100 8K .rat files hanging around

It would also be interesting to see the verbose output from your network render (how much data was loaded over the network – TBF texture, etc.).

Ok. That's correct then.

I will try your scene.
For the network logs. Do I just turn the verbosity to 1 on mantra?

What kind of network usage you get when you start rendering?

Could be also the combination of alembic file and UDIM textures what I'm experiencing.

im loading maps wit the names
map-0001.rat
map-0002.rat
etc.


correct?

So i did check your file and did some testing on it.
I created 10x 4k tif files in photoshop and converted them to RAT files using mplay. Then copy them to get in total 200 files.

Local render - from SSD
Network render - 1Gb network

Changed the render setting to PBR.

Here are some results
Local
Generating Image: ip (1280x720)
Plane: Cf+Af (16-bit float)
SampleFilter: alpha
PixelFilter: gaussian -w 2
VEX Type: vector4
Gamma: 1
Plane: Op_Id (16-bit float)
SampleFilter: closest
PixelFilter: minmax idcover
VEX Type: float
Gamma: 1
Plane: Prim_Id (16-bit float)
SampleFilter: closest
PixelFilter: minmax idcover
VEX Type: float
Gamma: 1
reading geometry from stdin
/obj/geo1/switch1 (2401 primitives)

Load Time: 6.08u 6.39s 0.64r
Memory: 100.59 MB of 100.60 MB arena size. VM Size: 626.46 MB
Ray Counts:
23,140,100 primary rays 25.11 rays/pixel
13,207,891 occlusion rays 14.33 rays/pixel
12,889,973 shading rays 13.99 rays/pixel
49,237,964 total rays 53.43 rays/pixel
Unified Cache: 9.38 GB of 31.98 GB used
In Cache Faults Generated Item Size
TBF Texture: 9.38 GB 1248200 9.38 GB 7.88 KB
Render Time: 2:37.98u 2.38s 2:38.83r
Memory: 10.29 GB of 10.29 GB arena size. VM Size: 11.17 GB

Peak Geometry Objects: 2

Network
Generating Image: ip (1280x720)
Plane: Cf+Af (16-bit float)
SampleFilter: alpha
PixelFilter: gaussian -w 2
VEX Type: vector4
Gamma: 1
Plane: Op_Id (16-bit float)
SampleFilter: closest
PixelFilter: minmax idcover
VEX Type: float
Gamma: 1
Plane: Prim_Id (16-bit float)
SampleFilter: closest
PixelFilter: minmax idcover
VEX Type: float
Gamma: 1
reading geometry from stdin
/obj/geo1/switch1 (2401 primitives)

Load Time: 6.55u 7.48s 0.65r
Memory: 100.50 MB of 100.50 MB arena size. VM Size: 626.46 MB
Ray Counts:
23,140,100 primary rays 25.11 rays/pixel
13,207,891 occlusion rays 14.33 rays/pixel
12,889,973 shading rays 13.99 rays/pixel
49,237,964 total rays 53.43 rays/pixel
Unified Cache: 9.38 GB of 31.98 GB used
In Cache Faults Generated Item Size
TBF Texture: 9.38 GB 1248200 9.38 GB 7.88 KB
Render Time: 1:58.41u 8.25s 17:10.59r
Memory: 10.29 GB. VM Size: 11.10 GB

Peak Geometry Objects: 2
The main problem is loading the textures from network at 5% of network capacity. Thats where i do see the bottle neck.


The scene you have is loading the lowest resolution of RAT textures, once you disable the MIP maps and force it to load the full res textures - thats what is happening on rendering the closeups of the giant you will see the extreme difference in render time, since Houdini cant load the textures faster then 5-10% of actual network limit and the CPU is idling (2-3% of CPU). Not mentioning that Mantra is dumping and loading the textures every frame (rendering giant as alembic cache).

And with MIP map enabled (loading the lowest resolution)
Local
Generating Image: ip (1280x720)
Plane: Cf+Af (16-bit float)
SampleFilter: alpha
PixelFilter: gaussian -w 2
VEX Type: vector4
Gamma: 1
Plane: Op_Id (16-bit float)
SampleFilter: closest
PixelFilter: minmax idcover
VEX Type: float
Gamma: 1
Plane: Prim_Id (16-bit float)
SampleFilter: closest
PixelFilter: minmax idcover
VEX Type: float
Gamma: 1
reading geometry from stdin
/obj/geo1/switch1 (2401 primitives)

mantra: Unable to load texture ‘ctemp/RAT/test/map.0000.rat’
Load Time: 5.61u 6.39s 0.61r
Memory: 100.61 MB of 100.61 MB arena size. VM Size: 626.46 MB
Ray Counts:
23,140,100 primary rays 25.11 rays/pixel
13,207,891 occlusion rays 14.33 rays/pixel
12,889,973 shading rays 13.99 rays/pixel
49,237,964 total rays 53.43 rays/pixel
Unified Cache: 98.89 MB of 31.98 GB used
In Cache Faults Generated Item Size
TBF Texture: 98.89 MB 13467 98.89 MB 7.52 KB
Render Time: 11.72u 8.26s 9.66r
Memory: 671.36 MB. VM Size: 1.55 GB

Peak Geometry Objects: 2

Network
Generating Image: ip (1280x720)
Plane: Cf+Af (16-bit float)
SampleFilter: alpha
PixelFilter: gaussian -w 2
VEX Type: vector4
Gamma: 1
Plane: Op_Id (16-bit float)
SampleFilter: closest
PixelFilter: minmax idcover
VEX Type: float
Gamma: 1
Plane: Prim_Id (16-bit float)
SampleFilter: closest
PixelFilter: minmax idcover
VEX Type: float
Gamma: 1
reading geometry from stdin
/obj/geo1/switch1 (2401 primitives)

Load Time: 5.46u 5.77s 0.62r
Memory: 94.54 MB of 96.53 MB arena size. VM Size: 620.39 MB
Ray Counts:
23,140,100 primary rays 25.11 rays/pixel
13,207,891 occlusion rays 14.33 rays/pixel
12,889,973 shading rays 13.99 rays/pixel
49,237,964 total rays 53.43 rays/pixel
Unified Cache: 146.89 MB of 31.98 GB used
In Cache Faults Generated Item Size
TBF Texture: 146.89 MB 19708 146.89 MB 7.63 KB
Render Time: 7.41u 7.48s 30.30r
Memory: 726.38 MB. VM Size: 1.79 GB

Peak Geometry Objects: 2

Hi,

We can take mantra out of the equation by running vexexec.

On Linux, you can flush the system disk caches using the drop_caches file in /proc/sys. This means that there's no network caching of the files.

On my system, when I run vexexec on the attached .vfl file, I get the following. Local textures take 7.26 seconds real time (off SSD), while networked textures take 11.34 seconds.

So, while network texture access is slower, it's less than 2x slower on my platform. What platform are you running on? What style of network file system are you using?

On my system, I'm getting about 40MB/s network download during the network test.

Output of testing:

% sync; echo 3 | sudo tee /proc/sys/vm/drop_caches; time vexexec timetex.vfl path /tmp/tex
3
0 percent..
10 percent..
20 percent..
30 percent..
40 percent..
50 percent..
60 percent..
70 percent..
80 percent..
90 percent..
Average color: {0.133253,0.139324,0.157896}
4.426u 0.262s 0:07.26 64.4% 0+0k 1102296+0io 478pf+0w
!sync; echo 3 | sudo tee /proc/sys/vm/drop_caches; time vexexec timetex.vfl path /mnt/hq/home/tex
3
0 percent..
10 percent..
20 percent..
30 percent..
40 percent..
50 percent..
60 percent..
70 percent..
80 percent..
90 percent..
Average color: {0.133488,0.139579,0.15837}
5.026u 0.337s 0:11.34 47.1% 0+0k 924032+0io 478pf+0w

i will have to check how to execute the file - first time i see something like that, i will post results in next post

we do run Windows, servers are connected with 1Gb link, im able to copy files at 90MB/s
Lame description of the network ….

I was not able to use the echo and time command, so i created bat with time %echo% before and after the vexexec.
I did change it to load 200 rat textures

Network 2.5s, ssd 1s

C:\temp\RAT\test>speed

C:\temp\RAT\test>echo 10:13:08.95
10:13:08.95

C:\temp\RAT\test>vexexec timetex.vfl
0 percent..
10 percent..
20 percent..
30 percent..
40 percent..
50 percent..
60 percent..
70 percent..
80 percent..
90 percent..
100 percent..
110 percent..
120 percent..
130 percent..
140 percent..
150 percent..
160 percent..
170 percent..
180 percent..
190 percent..
Average color: {2,1,1}

C:\temp\RAT\test>echo 10:13:11.22
10:13:11.22

C:\temp\RAT\test>speed

C:\temp\RAT\test>echo 10:13:36.55
10:13:36.55

C:\temp\RAT\test>vexexec timetex.vfl
0 percent..
10 percent..
20 percent..
30 percent..
40 percent..
50 percent..
60 percent..
70 percent..
80 percent..
90 percent..
100 percent..
110 percent..
120 percent..
130 percent..
140 percent..
150 percent..
160 percent..
170 percent..
180 percent..
190 percent..
Average color: {2,1,1}

C:\temp\RAT\test>echo 10:13:37.63
10:13:37.63

C:\temp\RAT\test>

martinkindl83
I was not able to use the echo and time command, so i created bat with time %echo% before and after the vexexec.
I did change it to load 200 rat textures

Network 2.5s, ssd 1s

C:\temp\RAT\test>speed

…

C:\temp\RAT\test>

The second time you run it, make sure to pass the path to the network file. For my system it was
% vexexec timetex.vfl path /mnt/hq/home/tex

Your test showed that the local path was nice and fast About 2x faster than my machine (unless you had run the test earlier and the file system had cached the data).

i did build the path inside the script and changed it for network test

after the test i notices that there was option to put the path after the command