I'm planning on installing a houdini workstation. Up until now, I've worked on linux with classic 32 bit machines.

Anybody has any experience with dual Opteron - houdini setup? I realise i'm trying to do something that is at the very edge of the available technology here. What I'm looking for is caveats, tips and tricks as I haven't bought any hardware yet. What motherboard, processor? Does the memory need to be of a certain kind? Do I need to fill the memory banks with equal sized memory sticks? Do I need more memory per se because of the 64-bit structure? I don't need to worry at all about the lower clockrates of opteron vs PIV? Any consequences for What has your preference or what would you buy if you had to buy a new system for houdini?

I have Gentoo installed on my current machine and I'm quite happy with it. If I was to consider another distro, it would probably be SUSE.

I can afford at this point in time to spent some time tweaking, setting up, compiling, … It's a jump in darkness a little bit, I'm guessing I'm looking for confirmation whether it can be done. A dual opteron SOUNDS very cost effective…

Thank you,

DRie.s

I'll let Paul jump in here about the niggly technical details - I know he's running a dual Opt and has had some troubles that *appear* to be related to it, although that's far from certain. Couple of quick obvious (to me ) answers:

You don't need more memory because of 64 bit - it's unrelated. 64 bit is an internal “bandwidth”, it doesn't increase the memory you need by two or anything like that. As far as type of memory, restrictions - all that will be dependant on the motherboard I'm sure - every motherboard has it's own mem restrictions, 32 or 64 bit. They're joined at the hip. Talk to whoever's selling you the mboard.

Go with whatever distro you're comfortable with - certainly I'm a big SUSE fan and highly recommend it, but if you know Gentoo already that would save you time.

As far as the entire 64 bit issue - I wouldn't agree that it's in any way a no-brainer to go dual Opteron. There's a whole raft of issues with 64 bit that will need to be dealt with. It depends on the sort of person yu are - if you like bleeding edge, are willing to put up with the(many) headaches, and get a personal rush from dealing with these sorts of things, then go for it. If you are thinking that it's somehow wasteful to be buying 32 bit, that couldn't be farther from the truth. In fact, you can get better dollar performance, as a rule, getting more, slightly older tech than going leading edge - that's almost a given in the computer world. You're not going to be leaps and bounds ahead of your competitors - but you'll have their endless gratitude being a guinea pig for them.

I may sound really down on it - I'm not, really. It will give you increased performance - but likely not as much as a couple years from now when the applications catch up. I'm considered somewhat of a hot-head when it comes to trying new things, but I'm not looking at 64 yet. Paul should be able to give you some front-lines commentary, though. He's a rebel.

Cheers,

J.C.

This thread is of great interest to me, we (UA's LPL Lab) will soon be getting a “computing cluster” composed of 16 dual 64 bit Opterons, 2G of RAM each, only the compute node will have disks, everything else diskless. It'll be running a 64 bit version of Fedora Core 2.

The primary purpose of this beast is to run Monte Carlo simulations (for gamma ray models of Mars) but I'll also be using it as a render farm for mantra and prman (maybe povray and BMRT too if they'll work).

Should be a blast!

I'll report any relevant experiences here and would love to hear anyones recommendations or tips.


–Mark

Hi John,

I'm afraid I know Paul all too well being a rebel and such, and I found out working with him that I can relate to his thinking. Our decisions have literally made our paths cross. Is it coincidence that he's going for an opteron system? Is Paul more bleeding edge than me? Of course I'm gonna go for an opteron! No doubt about it now! ;-)


I knew Paul had gone for dual opteron, but he's on a short break. He mentioned Rangi had such a system too, do you Rangi? I was just wondering if anyone else did, because you know John, these guys don't necessarily know why exactly they are going bleeding edge. I mean the theory and all, that's where people like you come in, right? Thanks for you point of view. Anyone else with technical details about Opteron?

The 64 bit side of it is a nice thing to look forward to in the future when the software, drivers and os are optimized for it. However the opteron is more about about the architecture used. It is a really well thought out design for multi processing that frees itself from common bottle necks.

The opteron's run just as fast in 32 bit mode with no performance hit and are at the same price performance point of xeon chips. I dont really see much with bleeding edge there just forward thinking which is re inforced by the fact that future multi core chips will fit into the existing socket/motherboards.

As for personal experience. The amd64 and 32 bit version of suse worked great along with windoze xp pro sp1. The 64bit drivers seem to be solidifieing right now.

ps. Bleeding edge would be buying a motherboard with pci express and DDR2. Opterons have been out for a while. :wink:

Ummm….

I dont really see much with bleeding edge there just forward thinking which is re inforced by the fact that future multi core chips will fit into the existing socket/motherboards.

Except there *are* apparently some software issues with it, stability-wise. That makes it potentially bleeding-edge in my books. Also - my understanding is that it's about trying to run them at 64 - don't buy something unless you're going to use it now. I'd be surprised if there was any tangible performance increases with dual Opts at 32 vs dual Xeons(which are really cheap). It sounds like you're buying something with potentially more problems and no serious advantage. For production - give me tried and true! When 64 bit becomes more standard, these older, earlier chips will lose support and drop off the map - just watch. As mentioned, unless you seriously get a jolt out of doing this sort of thing, I wouldn't bother.

Bleeding edge would be buying a motherboard with pci express and DDR2. Opterons have been out for a while.

Opteron birthdate: April, 2003

You're right that pice cards haven't been out there, but the tech market's been soft the last couple of years and pcie technology has been around longer, actually. The market wasn't ready for “a new standard” in cards - it is now. My next computer I'm getting in the fall is DDR2, pcie, and SATA all the way. The market for exactly what you've mentioned is starting to explode as of right now. Although I normally never do this, I've been waiting a couple of months for this to happen.

Cheers,

J.C.

P.S. Dries - yup I read your message and I'm not trying to talk you out of it… This is more to Warren…

We have many dual Opterons at work both as workstations and on the render queue. They're damn fast, and very reliable from what we've seen. However, R+H does have a systems dept that tunes the kernel and whatnot so the verison of redhat we're running (7.3) is _very_ different from the “real” Redhat distros.

Indeed, buying hardware that isn't going to be used is a waste, but it's good to hear that Peter is apparently using the opterons with great succes. Even though Houdini wouldn't be compiled for 64 bit, other software for the workstation could be, hence my choice for either suse or gentoo. I'm gonna put some questions out on the gentoo list and get their point of view, but generally, gentoo has been really quick in catching the very latest, even without using their experimental tree. My hope is it wouldn't be that painfull to run a linux workstation which uses 64bit software for the greatest part.
And if sidefx is gonna make a 64 bit version compiled for the new upcoming intels, does that mean that the Opterons will also run under 64 bit or will they be bound to 32 bit houdini versions?

pbowmar
However, R+H does have a systems dept that tunes the kernel and whatnot so the verison of redhat we're running (7.3) is _very_ different from the “real” Redhat distros.

Hi Peter,

Can you eleborate a bit on what those differences might be? Are they just tweaking various tunable kernel parameters or are there other kernel modules built into the kernel as well?


–Mark

I'm curious which flavor of the Opteron are being used and/or considered; we ended up going with the 244, the 248's are much more expensive it seems for the performance difference.

On the SESI dls web page, there's a few distro's for what appears to be 64 bit builds. e.g.:
Houdini6.1.442 for linux_ia64_rh2.1AW

What's the “ia64” mean? :?
And is “rh2” Red Hat Enterprise? ops: :? ops:


–Mark

Hey Mark,
Sorry, I can't give details, mainly because I don't know

Cheers,

Peter B

What's the “ia64” mean? :?

Hiya Mark,


Hmm… Isn't it stand for “Intel Architecture 64” or something? It's definitely Intel-based CPU, I am sure. Not sure if the IA-32, IA-64 is applicable to AMD-based CPU, tho…

-Alex

That's correct, and the “rh2” is indeed redhat enterprise, I'm quite certain, although I believe the more standard acronym for that is RHE…

Cheers,

J.C.

Hmm… Isn't it stand for “Intel Architecture 64” or something? It's definitely Intel-based CPU, I am sure. Not sure if the IA-32, IA-64 is applicable to AMD-based CPU, tho…

I thought that Intel Architecture 64 was just a rename of the amd's 64 technology because intel didn't want to loose to much face. So, does anyone know whether the houdini ia64 distro works on an opteron?

DRie.s

IA64 is the instruction set for the Itanium, which will definitely not work on an Opteron (or any other x86 chip, for that matter).

Intel's clone of the AMD 64-bit extensions is called “EMT” (extended memory technology). The latest Xeon CPUs have this, so they are essentially equivalent to Opterons.

I would be very interested to hear any performance benchmarks with Opterons vs. 3GHz Pentium 4s or Xeons. I will be doing a render farm upgrade soon and both are options.