Hi all, I've got a ~10 minute animated film project coming up and I've been researching how to render it all in a reasonable amount of time. I tried many of the common commercial options and all shared two problems: they get real expensive real fast and a lot of the process is out of your control.

I was about ready to buy some PCs to create a mini-farm at home, but I had the thought to revisit AWS and Azure. I won't go into exhaustive detail, but suffice it to say that AWS's Thinkbox and Nimble Studio, while very cool and useful, are way overkill for just one guy. I ended up rolling my own solution on Azure. Azure is dirt cheap and not too hard to use. Here's the workflow I settled on after several weeks of on and off testing:

- Install CentOS 8.2 on an Azure VM. CentOS has no hourly fee so you'll just pay for the vCPU time, not the OS.
- Install a desktop environment (although technically you don't really need it): I tried Gnome, Xfce and KDE and KDE was by far the best.
- Install whatever CG stuff you need. I put Maya, 3Delight, Arnold and Houdini Engine.
- Set up the azcopy utility on my local Windows machine and the Linux VM. This lets you easily transfer files to and from Azure blob storage.
- For the final test I created a short animation in Solaris, exported to USD, and uploaded the whole project folder structure to Azure.
- From there, download the project to the VM and run husk to render it in Karma.
- Finally, azcopy up to the blob and back down to the local machine.

Yes, this is a lot of steps and was a bit of a pain to get set up. Linux is never easy to deal with and all these industrial strength cloud services are designed for IT pros, not your average guy off the street. Still, Azure is pretty easy to learn and there are a lot of tutorials and good documentation.

In the end this render probably cost around $1 for VM time and a few cents for storage and bandwidth and such. Nothing fancy, just something quick and noisy as a proof of concept. But, once I get up to rendering 15,000+ frames for the finished film and accounting for longer render times for more complex scenes I'm still going to be paying like a 10th of what I might elsewhere. Plus, I have total control of what is getting rendered and when, what the status of uploads and downloads is, etc.

I've never seen anyone talk about this solution for indie artists so I wanted to write about it. If you have the patience to deal with the setup time then I highly recommend it. Seems extremely robust, reliable, affordable and scalable.

UPDATE: Tested out a spot Windows VM on Azure. The idea behind these is you get a huge discount (as low as 10 cents per hour) but can be kicked out of the instance at any time. Here's a 72 frame, 500MB sim that ended up costing about $0.60 to render. Only got kicked once, before the very last frame. Easy enough to log back in and finish just the frames needed. Loving this system!!!



Original Solaris test:

This is fantastic! (jumped here from the link you placed into my question)

I need to wrap my head around Karma before doing this (or I could just run it in Mantra...)

For most of the work I do I get reimbursed for ChaosCloud or whatever service but it would be really nice to have more control and not have to shell out the initial cash to render (and possibly charge a small fee myself and just pocket that... everyone would save some money!)

I've setup a Deadline/AWS pipeline for my day job studio but am always on the hunt for something more manageable for my personal/freelance work.

Thank you so much for sharing this!!

Hi, have you tried also to render something in Redshift in one of the VMs with GPUs?

We have an heavy scene to render and we're looking at the diffrent options that are on the market

RiccardoT
Hi, have you tried also to render something in Redshift in one of the VMs with GPUs?

We have an heavy scene to render and we're looking at the diffrent options that are on the market

I haven't tried GPU. The Azure NCsv3 series looks ideal for GPU rendering, V100 GPUs which are OctaneBench 350 or so. They're pricy per hour, but maybe so fast that it ends up being the same cost as rendering the scene more slowly on CPU...

BrianHanke

RiccardoT
Hi, have you tried also to render something in Redshift in one of the VMs with GPUs?

We have an heavy scene to render and we're looking at the diffrent options that are on the market

I haven't tried GPU. The Azure NCsv3 series looks ideal for GPU rendering, V100 GPUs which are OctaneBench 350 or so. They're pricy per hour, but maybe so fast that it ends up being the same cost as rendering the scene more slowly on CPU...

It's a scene with a lot of reflections and refractions, we tested it with mantra and it was a pain just get a few buckets, then we switched to Redshift and the results were there and we could actually do lookdev but now we have the problem that the scene it got so heavy that we are running out of VRAM on some of the cards that we have and we are looking on cloud rendering

We'll look at the NCsv3, thank you

Have tried both AWS and Azure?
Which one is easiest to setup?

So far we found this https://aws.amazon.com/getting-started/hands-on/studio-in-the-cloud/ [aws.amazon.com]

We don't need something so complex but a way maybe throught a VPN to connect our internal network to the one hosting the space will be a nice plus to have

RiccardoT
Have tried both AWS and Azure?
Which one is easiest to setup?

I find Azure much easier to use. I feel like AWS is designed more for IT pros. The Nimble Studio and Thinkbox offerings from Amazon are way overkill, for medium to large studios only. An individual or small group is much better off setting up their own VMs.

Try out a Windows VM with V100 GPUs and see how it goes! (Or if you're good with Linux that works too.) At most it will take a few hours and cost maybe $10-15 max.

BrianHanke
Try out a Windows VM with V100 GPUs and see how it goes! (Or if you're good with Linux that works too.) At most it will take a few hours and cost maybe $10-15 max

Fantastic, I'll look into it in the weekend

Just reviving this thread in hope of hearing any new conclusions or experiences. I am looking for a solution for scaling Vray, ideally spinning up at least 10 high core count machines for quick bursts of jobs. We do use redshift locally and had some experience with 3dl so if any other solution is better/stabke we are quite open to try that as well.

For months now I've been using my Azure setup for rendering all animations and high resolution stills. Solaris and Karma are a perfect fit for it. Just export the scene as USD, update paths to assets as needed, and render from the command line. I can do it all from PowerShell on my laptop.

- Upload all assets to Azure blob storage via azcopy.
- Create a 96 vCPU VM from my custom Debian image with H19.
- Start the VM, log in via ssh from my laptop.
- Download the assets using the Linux version of azcopy.
- Sort out paths if needed.
- Render using Karma.
- Upload finished frames via azcopy and download them to my local machine.

A 96 vCPU Linux spot instance on Azure is less than $1/hour. I've been doing some character lookdev recently and a 2k frame takes Karma about 2 hours to render on my laptop. I've just been sending it off to Azure instead: 15 minutes total time and final cost of less than 50 cents.

It's the perfect solution for me and I couldn't be more pleased with how it's all working. Extremely affordable and very flexible. I've also used Clarisse and Maya/Arnold in the same setup and both worked great.

Hey Brian,

thank you for sharing your insights, they are immensely helpful.

Thanks for sharing the experience Brian. Was thinking of going with aws deadline first but maybe this is easier. Also the azure “batch” systems seems quite enticing.

Good luck, let us know what you end up doing! For some reason I find AWS very hard to use, but that might be just me...

BrianHanke
Good luck, let us know what you end up doing! For some reason I find AWS very hard to use, but that might be just me...

How did you manage the houdini licenses?
Is Azure cheaper than aws? Do you think running on aws by deadline would be easier to manage the ongoing renders?

guilhermecasagrandi
How did you manage the houdini licenses?

I use login licensing. It's a lifesaver for stuff like this. I have a script that makes sure everything is tidy before starting rendering: sesictrl login, then setup Houdini's paths and variables and such. (Old school license server style like Clarisse works too, but you're playing with fire if anything happens to the VM.)

guilhermecasagrandi
Is Azure cheaper than aws? Do you think running on aws by deadline would be easier to manage the ongoing renders?

I think AWS, Azure and Google Cloud are all around the same price. They all have different VMs, Google has more cores on some, Azure is lacking GPU options, but in the end the processing power costs about the same.

AWS and Deadline isn't easier for me since I do not understand Deadline AT ALL. Custom scripting and controlling renders via SSH makes more sense to me somehow. But, for big jobs I guess you'd have to use something like Deadline to keep organized...

BrianHanke

guilhermecasagrandi
How did you manage the houdini licenses?

I use login licensing. It's a lifesaver for stuff like this. I have a script that makes sure everything is tidy before starting rendering: sesictrl login, then setup Houdini's paths and variables and such. (Old school license server style like Clarisse works too, but you're playing with fire if anything happens to the VM.)

guilhermecasagrandi
Is Azure cheaper than aws? Do you think running on aws by deadline would be easier to manage the ongoing renders?

I think AWS, Azure and Google Cloud are all around the same price. They all have different VMs, Google has more cores on some, Azure is lacking GPU options, but in the end the processing power costs about the same.

AWS and Deadline isn't easier for me since I do not understand Deadline AT ALL. Custom scripting and controlling renders via SSH makes more sense to me somehow. But, for big jobs I guess you'd have to use something like Deadline to keep organized...

Thanks for the answers!

You said you are using login-based licenses. Do you use indie or FX licenses?
AWS Thinkbox has usage-based licenses, but I don't think Azure has. How do you scale render nodes?

I'm using Houdini Indie. The included 3 (4?) Karma licenses are plenty for the stuff I do.

If I wanted to scale up for a big project I would get monthly rentals of whatever renderer I was using. I did a mini version of this with Clarisse, renting a CNode for a month. They use old school activation licensing so that almost led to disaster when the VM I had it installed on randomly failed. Isotropix was nice to me and issued a new hardware ID, but that's why I prefer login!

BrianHanke
I'm using Houdini Indie. The included 3 (4?) Karma licenses are plenty for the stuff I do.

If I wanted to scale up for a big project I would get monthly rentals of whatever renderer I was using. I did a mini version of this with Clarisse, renting a CNode for a month. They use old school activation licensing so that almost led to disaster when the VM I had it installed on randomly failed. Isotropix was nice to me and issued a new hardware ID, but that's why I prefer login!

I'll have to check how many karma licenses are available for each HIndie one. I thought there were just 2 for each.
For a heavy scene, 3 render nodes don't change that much for fast turnarounds... I'll have to check with sesi for options.
I didn't do much math on the numbers thinkbox is providing, but I'm thinking 60 cents/machine-hour for a houdini engine license seems too much for a small studio.

Yeah my impression is that Thinkbox UBL is VERY expensive. You pay for convenience. An Arnold license is $45 for a month, that's 720 hours. 1000 hours UBL for Arnold is $660. I don't get it... But, I shouldn't comment too much on what happens when you try to scale Azure up with lots of VMs and licenses since I haven't done it. In theory it works!

BrianHanke
Yeah my impression is that Thinkbox UBL is VERY expensive. You pay for convenience. An Arnold license is $45 for a month, that's 720 hours. 1000 hours UBL for Arnold is $660. I don't get it... But, I shouldn't comment too much on what happens when you try to scale Azure up with lots of VMs and licenses since I haven't done it. In theory it works!

Brian, how did you manage to set sesictrl on a headless machine for login-based-licensing?
I can't get It to work....

Here's the script I use on Linux VMs: