Dave_ah
May 10, 2003 06:27:51
I am not sure where the fault lies. My lack of experiance with Houdini POPs or how the system is implemented.
I designed a nice working popnet for doing rain splashes. The particles fall, get repulsed, collide with ground, split. Everything is grouped.
The POP SOP is reading the rain popnet then Delete SOP is deleting everything but split particle (the splashes). The Geometry ROP is reading the Delete SOP and exporting the particle points as bgeo sequence.
The popnet was designed so it could be adapted for the actual scene.
The test scene had 1000 poly mesh emiiter, and scene is 100 frames.
The actual project scene has 5000 poly emmitter that is physically quite larget, and scene is 1307 frames.
In both test and project scenes the average constantrate is about 200,000 with Radom Surface emmission.
So only difference between test scene and project scene is size of emitter and frame count, everything else is same
The particle baking through Geometry ROP had good render times in test scene (100 frames).
In project scene I set to to bake popnet into gemetry using same Geometry ROP. And 5 hours later Geometry ROP still reports “Birthing Particles” Frame 1.
Now sence I had good per frame cooking time in test scene, I logically expected the same time to be in project scene. Sence they are not, I cannot logically predict baking times for heavy particles scene.
I don't like the way this system works. Its very very slow and unpredictable. 100 frame scene has tottally different per frame time then 1307 scene, with everything else being same. How's that?
Our Maya guys are working with similar number of particles and they are working much faster, using similar hardware. Whats up with Houdini? I mean 200,000 emission rate is not that high. Certainly does not warrant hours per frame baking times. What the heck is going on? I am behind scheduele and Houdini is utterly failing me.
What happens when scenes have millions of particles?
Dave
Dave_ah
May 10, 2003 12:10:48
Heres what I trully do not understand, and I think it really is fault of how Houdini works.
It takes 15 minutes to cook the popnet, when I am in Object and viewing scene through camera.
But when I set up GeometryROP to output displayed popnet geometry as read through POP SOP, the cook time failes. After 2 hours its still on frame 1, cooking the popnet. From looking at the messages, it looks like the cooking is going around in circles, the same cook messages are displayed. It makes no sence. I think this is a QA issue. Sorry for the critic.
Dave Rindner
Dave_ah
May 10, 2003 13:27:17
It goes without saying how much I am disappointed and agast at performance of Houdini's particle systems when dealing with large number of particles. I expected a lot more. When cook times go into HOURS per frame, with about 100,000 emission rate, its a signal that internally Houdini is badly designed. With constantrate set to 50,000, it takes more then an hour to make any change. Even with rate to 2000, it takes 5 minutes. This is disgusting. How can anything be done with such low level of productivity. Mere termination of cooking, by ESC key, takes more then 20 minutes. For 20 minutes the user has to stare at message “Waiting for popnet cooking to terminate”. THis is unreasonable, and is incomptabile with production deadlines.
I can understand not being able to achive an effect due to experience. But what I am experiencing are clearly badly designed or poorly implemented features of the application.
Such a stark contrast to SOPs.
I doubt I can trust POP to deliver for me on next project.
Dave R
EigenAlex
May 10, 2003 13:59:04
David, if it's not NDA, can you please tell me a little bit about your pop network?
–Is it possible for you to break things into layers instead of emitting 100k particles *all at once*?
–Are you birthing the particles from Location POP or Source POP?
–Are you using Constant birth-rate or Impulse birth-rate? What are its value? Are they birthing on every frames or only on a specific range of frames?
–Did you create any sort of complex attributes and calling? Expressions and nesting?
–What kind of collisions do you have? What are its oversampling and bounce accurcy?
–Did you set the popnet's oversampling too high?
–In your Geo ROP, did you turn on “Initialize Simulation SOPs” and is that necessary if you turned it on?
–Are you rendering out *only* the particles point or do you have things copied to those point and then render things out?
Just that you mentioned that you set the “constant rate” to “50 000”. Sounded like it would have about 150 000 to handle by the time it reaches frame 90…
HOpe that helped a little.
Alex
EDIT: oops… I skimmed too fast.. didn't quite read your first post…
Dave_ah
May 10, 2003 14:18:22
–Is it possible for you to break things into layers instead of emitting 100k particles *all at once*?
–Are you birthing the particles from Location POP or Source POP?
Source POP- A 16 X16 poly ,mesh, with Random Surafce option.
–Are you using Constant birth-rate or Impulse birth-rate? What are its value? Are they birthing on every frames or only on a specific range of frames?
I am using Constant rate with rate controlled by Noise CHOP. So it is not consistent.
–Did you create any sort of complex attributes and calling? Expressions and nesting?
I have Properties POP for Mass and Drag, as well as Group POPs, so that Delete SOP (appended to POP SOP) deletes the rain, but keeps the splashes , which are split of from collided rain drops.
–What kind of collisions do you have? What are its oversampling and bounce accurcy?
One collission with 5,000 poly mesh object (the road). One Attractor SOP (negative) to repulse particles from car.
–Did you set the popnet's oversampling too high?
1 as is default
–In your Geo ROP, did you turn on “Initialize Simulation SOPs” and is that necessary if you turned it on?
Yes it is turned ON, and yes its neccessary as I usually work with update set to NEver or Change.
–Are you rendering out *only* the particles point or do you have things copied to those point and then render things out?
Just particle points. ANother FileSOP in another Geo node is reading the bgeo sequence for instancing, so it is unrelated to pop cooking.
The point is that cooking never gets done. To me it looks like its going around in circles performing same POPs over and over again.
I simplified the project by dividing it into three parts each truncated, from original, into 500 frame. No change. I thought that the particles dynamics are affected by frame counts. So if I could write out three pop bgeo sequences, 0-499, 500-999, and 1000-1307, combine them in one folder then read them using File SOP in 1307 frame scene. But cook times are still ridiculous.
This is a new long scene, and I am using same POP as I did for short scene (100 frames) for which I had excellent results. The water splashes look like splashes when rendered, and rendering times are good (Mantra).
But as soon I changed time project to 1307 frames the cook times went through the roof. I am applaled. This means that basic pre-production R&D cannot be trusted when applied to project scenes.
DaveR
EigenAlex
May 10, 2003 14:58:16
Please bear with me here. I am still trying to guess at what the problem is.
Couple more questions.
So you don't really have any expressions in your POP network, is that correct? As far as the “Initialize Simulation SOPs” goes, I am still puzzled if this is even necessary at all as you are not doing any sort of dynamics simulations (i.e. RBD). Could you try turning this one off?
So the total number of frames you have to render is 1307 frames for the project right? Here's the thing, by the time you reached frame 1307, you may have 87133 particles at frame 1307 if constant rate is set to 2000 (2178330 if at the rate of 50000 – I believe this is quite a ton for the kind of pipeline you have) – presuming that you're not killing your particles until at SOP level.
Also, why do you use Delete SOP to remove your points instead of using Delete POP? The way I am looking at it is that when you use Delete SOP, it's only deleting points *after* you finished your particle, hence you have massive amount of particles for Houdini to calculate. If possible, try deleting the unnecessary particles within the POP network?
ALso, you might want to look into the Performance Monitor to see which OP is taking the longest time to cook, and then see if you can find a work around to that OP.
Can you pretty much kill all particles that you know you will never see again?
Here's a couple of questions from Jim P.:
–Do the first 100 frames still work?
–And any frames after that are too slow? or is frame one cooking too slow
–when the particles hit the road do they die? (or die after a couple of bounce?)
–on the actually POPnet there is a check box that says remove unused points, is this turned on or off for you? I turned off remove unused points and even thought particles were dead, they wouldn't go away
Dave_ah
May 10, 2003 16:09:12
There is no Delete POP in 5.5. Thats why I use Delete SOP. I am not a V6 user yet, if DeletePOP is new to it.
I do not have Initialize SOP in GeoROP.
I dislike the system becouse it is self-contradictory. For example, I was advised to use Cache. But when I do, Houdini becomes unstable.
So whats the point of caching, if the app crashes.
You are incorrect in your assumption that by end of scene I have overload of particles. The lifetime and killing of particles at splits, keeps overall popnet weight hanging around 225,000 visible particles at any one time. The performance is abysmal.
Dave R
Dave_ah
May 10, 2003 16:48:48
I want to give you an example, of why I am frustrated by how pops work.
It takes about an hour for net to cook, an appalling amount of time.
Now the POP SOP reads it and appended Delete SOP cleans it.
In Geometry ROP I have Initialize Simulation SOPs turned off.
So before I can write the pop points to disk I have to let the system cook in SOPs, then go to Output and press Render or GeoROP.
OK so I let the popnet cook, and wait for an hour or so.
Then I go to Ouput and I press render. Then I remembered that I had forgotten to change the point groups which DeleteSOP has to cleanout. A minor oversight. So I ESC the render. Then I drop back into Geometry to change the setting in Delete SOP. Tragio-comically I had also neglected to check that update had to be set to NEVER or Changes. It was Always at that time. So as soon as I dropped back in Geometry, Houdini started to recook the popnet. I immidiatly pressed ESC to cancel re-cook. To late. All the previous cooking was wiped out. So few minor oversignts and whole hour wasted. Actually two, since I have to recook. Now I take responcibility for not noticing minor details in middle of the night after 48 hours in the shop. But that incident shows how user unfriendly the particle system is implemented. The way that you guys have implemented the system, a user has to be PERFECT, to use the app efficiently.
Dave R
EigenAlex
May 10, 2003 17:45:34
My apologies. I meant KIll POP.
I've chatted with Jim P. for a bit earlier. Your Attractor POP is what seems like the case of a major slow down. If your car is about 3k point count, that's that'd be equivalent of a few k of Attractor POP put together. Combined with couple hundred k particles, you can expect the cook time to be crawling I am sure.
For crying out loud, the Cache SOP will cache things into your RAM. I don't usually use Cache SOP unless it's for something smaller and for quick and dirty previews that I know my RAM will hold. In your case of particles, render it out to disk. Cache SOP will take up all your system's memory resources.
Open up the Performance Monitor, check which OPs has the longest cook time, and find a different solutions for it. Open up a clean sessions, slap a couple of dummie geos in there and an attractor POP, open up the performance Monitor and see which OP took the longest to cook. I think you'd get an idea.
THe ALWAYS, NEVER, or CHANGES are usually used for the cooking in the viewport/Network Editor, it should not affect how it get cooked when you cook the geometry out. Regardless of what you do, it will always have to cook prior to rendering.
Oh, one more thing, I am by no mean perfect. I am the least perfect of all Houdini users in the world. Currently, a couple of us are trying to guess at what you did and trying to guess what the solutions may be.
Hopefully that there will be more experience people as well as SideFX personnel to clear things up on Monday.
Dave_ah
May 10, 2003 18:13:15
The attractor is only few hundred points, and is roughly shaped like top of the car. It has to be the way it is, at least given my level of experience with Houdini.
Monday is when the job is due, so no need to bother others.
I'll be happy to e-mail you the scene, once I have the job finished.
So cache is for simple and light popnets. Thats kind of defeating the pourpouse sence it is the heavy scenes that need it.
I do want to know. Does length of scene effect cooking time. So given same popnet, should the cooking take significantly longer. If so is the calculation time, scale linearly or on a log curve.
Dave R
EigenAlex
May 10, 2003 18:33:20
Hmm… I highly doubt that the length of the scene affect particle cook time. However, I do believe that as the time drags on, and if you don't kill the unnecessary particles, it will accumulate over time, hence larger memory footprint and slower cook time.
Please do email me what you did if it's nothing violating NDA.
calling@poh-yee.comI cannot guarantee that I know how to solve the problem, but I can definitely take a look and see what I can do to help you out.
Dave_ah
May 10, 2003 20:06:24
You mentioned the Always, NEVER, and CHANGE have no effect and that cooking will always occur prior to rendering. Not so, at least not with GeometryROP.
If Initialize Simulation SOPs is OFF, then the user has to manually initiate the cooking proccess for popnets. If a cook is canceled and GeoROP is run, after cook is cancelled, the ROP runs its course and outputs empty geometry.
In my popnet. If Initialize option is turned ON and ROP is rendered, that 1st frame initialization NEVER finishes. I tried that early and have come back 5 hours later with the BIRTHING PARTICLES stuck on frame 1. You could sit there and watch it running through various pops, but it never gets beyond frame 1. Matter of fact it jusr re-cooks same pops over and over again but doesnot get beyond frame 1.
Now with Inialize turned off, I have to let the cook happen once at frame 1, in Geometry. That takes about an hour. After that cook is complete, I can go to ROPs , and press render. It writes out the sequence without that long time initial cook.
This is how it works, and I dislike this proccess. It is unintuitive, and time wastefull.
How do users deal with millions of particles that interact with geometry?
I did my experimetns with low particle counts. But as soon as I increased counts to production requirements, HOudini choked. Very disappointing.
DaveR
Dave_ah
May 11, 2003 09:10:44
Alex thank you for your help. The job is rendering, finally. Initial composite tests with finished passes look good. The ending cut for the add, which is already in post, looks damn fine. The Inferno girl ran some lens distortion through alpha, so my splashes look far more real then my pre-comps in Shake. Its amazing what Inferno can do with imperfect 3d renders and good fresnel alpha.
I did not realize that dynamically the rain splash effect is so DAMN TOUGH.
My solution, is unelegant, and inefficient. Its brute force method. Without using particles, and lots and lots of them, I can't see another way of covering large swaths of road and sidewalk with splashed water. The director wanted those funky bowl shape splashes, so I had to use particles and metaball passes. And he wanted it to cover large area, and e dense. Just to bake the particle points, I had to break the simulation into thinned out popnets.
I aplogize for my trepidation, if they caused you any concern. I just set a personal record of 72 straight hours in the office, with only quick runs to muy apt for a change. This is the most intense production I have been involved in to date.
I had a phone interview, went so so, with D2 people, and told straight up. I want to come to D2 and be Houdini modeling TD. God willing it may happen, it may not.
See despite myself I love this app. I wish it were 10X easier to use and 100X easier to learn. Either way I think I am hooked on it. For now

If we meet at SIGG in SanD, let me buy you a can of Fosters, if thats your thing.
Dave R
EigenAlex
May 11, 2003 15:30:22
Hi David, I am very glad to be able to at least help out – albeit it really wasn't much in my part at all. I must credit it to Jim P. for pointing things out. The longest I've stayed up was probably 48hrs. FX works does get highly technical, as I've learned in recent year.
Keep on going. You will have plenty of moment of “click”. Best of luck with DD! Please remember me when you're in there or the likes.
Shannon Gold
May 11, 2003 17:16:39
Yea, I've done a couple 72hr sessions myself. The brain fog is no fun.
Could one of you guys summarize the solution to David's cooking problem. You've got my curiousity peaked here.
David, good luck with D2. Get ya working back in the states.
Dave_ah
May 12, 2003 05:10:30
Hi Shannon
I'll try to explain the big picture. Starting with live action.
The car spot calls for car to drive up to camera, the camera turns and pans, until we see the side profile of the car, with office buildings in BG, then car drives away. Cut to ending shot where the car and camera stand still.
Thats the live action, that we got after TC transfer.
The scripts and boards call for everything to happen while heavily raining.
Heres is the kicker the rain cannot touch the car, but has to curve around and flow along it with an offset. The splashes have to be created also, as principal photography , with wet pavement, did not have them.
We deivided the job into two parts. My team mate was going to create rain streaks, and flow, and my task was to create the splashes. It turned out that splashes are more complex.
The rain streaks, we did in Maya using curve flow arrays, that were then masked in AE to get rid of errand particles.
Becouse the rain was so dense and amorphous, we could do splashes in Houdini, and when two are comped you can't tell the difference.
Heres how I did the splashes.
First we matchmoved the motion in Maya. The camera motion was from Mark Roberts MCC rig, and came as Maya file.
I built a digital version of the car, and match move geometry (the road, sidewalks, and building). Another animator then matchmoved those models, and we had a good refence Maya file.
Using MEL scripts from Steve Ong and Alex, that I got from odforce, I exported camera motion and car motion, and geometry out of Maya, and rebuild the matchmove in Houdini. That was easy and fast.
The car had to repel particles, so I could not have splashes happening in repelled volume. I built simple shell and boxes and used them as repulsion volumes. To create the dynamics I started by defining the rain.
The rain particles fall, get reppelled from the car, eventually collide.
When the rain particles collide they split multiple times. I defined the hit points and split groups. The split particles are subject to forces and bounces, to emulate splash dynamics.
I then read the popnet into POP SOP enmassee. To the POP SOP I appended DeleteSOP which was deleting all point groups except splash group and raindrop hit points.
Becouse this was very heavy and slow cook time, I baked the particle into *.bgeo sequence using Geometry ROP reading the Delete SOP.
Once the baking was done, which is a time consuming proccess on par and exceeds render times, I read the bgeo sequence using FileSOP.
Now I could slide back and forth in time without waiting for popnet cook.
To the FileSOP I appended two DeleteSOP as branches, not one after another.
The fist Delete SOP left only hit points, the second one left only the splashes.
This was possible becouse bgeo had remembered the point groups. Very convenient on party og Houdini!
THe hit points were then used as template for Copy SOP, and instance object was a circle primitive. So now I had this appearing and disappearing circles, based on raindrop hit points.
The splash points were used as template for another CopySOP and simple 3 poly primitive was used as an instance.
The third CopySOP used splash points as template and metaball as instanced objects.
Three passes were then rendered.
The ‘Splash Circles’, “Splash Primititives”, and Splash Droplets".
The three passes were then post -procceced in After Effects and Shake to make them look more liquidy, and to adjust the alpha fall off.
The passes were then passes to Inferno at post facility. There I reccomended to director the following compositing method.
Duplicate the telecine (refered to as A-Copy)
Substitute alpha of splash and rain passes as alpha of the a-copy.
Run distortion filters to emulate refraction and light multiplication as seen through rain drops and splashes.
Composite that distored a-copy (with rain and splash alpha) over original telecine with screen or color dodging transfer mode.
Dave Rindner
Dave_ah
May 12, 2003 09:04:58
One thing I forgot to mention in regards to cooking.
I had long cook time and needed a 1307 frame sequence of bgeo file.
I computed first 300.
Then duplicated the folder. Renamed the numbers, then moved the renamed files into original sequence folder.
Did that until I had 1307 bgeo files.
Dave Rindner
Shannon Gold
May 12, 2003 13:02:41
So, the cooktime that seemed to loop frame 1 was due to the number of frames in a sequence which is extremely particle and dynamics heavy.
And breaking into smaller sequences is the workaround. Or am I misunderstanding you?
Wow, you nailed it with your compositing method. Managing the shading of that effect would have been another nightmare. Good call.