I'm running into maybe a workflow issue regarding rendering lots of instances.

In the attached file, I'm able to render out 100k instances with this workflow:
A. PACKED DISK PRIMITIVES

1. Save out a bgeo.sc of my geometry.
   
2. Load it as a Packed Disk Primitive.
   
3. Use Copy to Points.
   
4. Render.
   

• It finishes at about 40 secs on my machine.
  
• Uses up to about 2.6gigs of memory.
  

But, if instead I use an alternative workflow like so:
B. PACKED PRIMITIVES

1. Use Copy to Points.
   
2. Enable Pack and Instance (Display as Centroid).
   
3. Render.
   

• It now renders at about 2 mins.
  
• Also uses around 2.6gigs of memory.
  
• It is roughly the same if I skip #2, but use a Pack SOP before Copy to Points for packing.
  
• Also, something quite interesting, if I use .bgeo (uncompressed).. it also renders very slow.
  

Summary for 100k instances:
A: 40s | 2.6gigs
B: 2m | 2.6igs

I also tested out 1 million instances to stress test this:
A: 2m 30s | 18gigs
B: 15m 30s | 18gigs (!!!)

Questions:

1. I would stick with Workflow A, but I would also like to understand what could be causing the large discrepancy in render times from Workflow B?
   In my observation, it would seem like Houdini is having a hard time loading the Packed Prims to memory (thus taking too long to prepare the render).
   As opposed to using the Packed Disk, where it seems to be able to load the prims only as needed.
   
2. And lastly, are there alternative workflows that would be much more appropriate for rendering lots of instances?
   

Machine Specs: i7-3.4GHz | 32GB Memory
H16.0.600 Indie

(semi-related thread where the scene originated) [sidefx.com]

You need to feed your filecache nodes a packed prim? Then it loads in under a minute, which is still long IMO but not the 15 you're getting.

Thank you for the suggestion Simon, but sadly it did not improve the render time on my machine. Actually it is very close to Workflow B's results.

Workflow C @ 100k instances:

1. Pack your geometry
   
2. Save it out as a bgeo.sc
   
3. Load it as a Packed Disk Primitive.
   
4. Use Copy to Points.
   
5. Render.
   

• Render Time: 1m56s | 2.6gigs
  

One would think that using Packed Primitives directly should be faster as it probably does not entail the cost of reading data from the disk (as Packed Disk Primitives). Or at least the render times should be close or the same for both workflows.

Added PolySoup into Workflow A and got a bit of improvement:

D. POLYSOUP > PACKED DISK PRIMITIVES

1. Convert your geometry to a Polygon Soup. (Convert SOP)
   
2. Save out a bgeo.sc of the geometry. (FileCache SOP)
   
3. Load it as a Packed Disk Primitive. (FileCache SOP)
   
4. Use Copy to Points SOP to instance.
   
5. Render.
   

100k instances:

• From 40s to 38s
  
• From 18gigs to 17.32gigs
  

1M instances:

• From 2m 30s to 2m 09s
  
• From 18gigs to 17.32gigs
  

Reading through the docs again, I found this part:

Whereas Houdini must write the entire geometry for any in-memory geometry into the IFD (the scene description file it sends to Mantra), for packed disk primitives it simply writes the reference to the file on disk. This can make IFDs much faster to generate and smaller on disk for very large/complex scenes.

This probably explains why Packed Primitives seem to eat up much larger memory.

Packed Primitves as described by the docs are “in-memory”. So they do get saved to the IFD.
As opposed to using Packed Disk Primitives, only a reference to a file on disk gets saved.

This is making much more sense now