Solaris rendering on the cloud

   12479   47   11
User Avatar
Member
433 posts
Joined: April 2018
Offline
I've had a chance to stress test my Ultimate Rendering Script a bit now and it's working very satisfactorily. As close to pure rendering automation as I've been able to get. Costs are amazingly low when using Linux and spot VMs. Fun but useless fact: you can even manage the renders from your phone with SSH!

First script is PowerShell to repath a scene file from Windows to Linux, package all the assets, upload them, create a VM, and open an SSH session.

Script number two is run in said session. It downloads the project, extracts it to the right folders, kicks off a render, uploads the results and shuts down the VM when done.

This particular script is for Maya, but it should work similarly with any other DCC or renderer.

I censored a few potentially sensitive Azure tokens and paths. Feel free to ask about anything and share your solutions too!

PowerShell

# define variables

param([string]$project)

$root_path="c:/onedrive/main_projects"
$token="##########################################################"

$geo=$root_path+"/geo/commercial/"+$project+"/*.usd"
$geo_dest=$env:TEMP+"/bhgc/"+$project+"/geo/commercial/"+$project+"/"

$tex=$root_path+"/tex/commercial/"+$project+"/*.*"
$tex_dest=$env:TEMP+"/bhgc/"+$project+"/tex/commercial/"+$project+"/"

$scene=$root_path+"/scenes/"+$project+".ma"
$scene_dest=$env:TEMP+"/bhgc/"+$project+"/"

# make destination folders

if (-not (Test-Path -LiteralPath $geo_dest)) {
	mkdir $geo_dest
}
if (-not (Test-Path -LiteralPath $tex_dest)) {
	mkdir $tex_dest
}

# update paths in project file

(Get-Content $scene).Replace("C:/onedrive","/home/brian") | Set-Content $scene

# copy files to assembly folder

copy-item $geo $geo_dest -Recurse
copy-item $tex $tex_dest -Recurse
copy-item $scene $scene_dest -Recurse

# archive scene

$7z=$env:TEMP+"/bhgc/"+$project+"/"+$project+".7z"
$7z_contents=$env:TEMP+"/bhgc/"+$project+"/*"

7z a -t7z $7z $7z_contents

# copy archive to Azure

azcopy copy $7z "######################################################"

# delete local archive

Remove-Item -Recurse -Force "$env:TEMP/bhgc/*"

# create VM

$template="C:\onedrive\Scripts\azure\vm_templates\tem_lin_96_spot.json"
$param="C:\onedrive\Scripts\azure\vm_templates\param_lin_96_spot.json"
Connect-AzAccount -TenantId #####################################
Write-Host -ForegroundColor Green "Deploying..."
New-AzResourceGroupDeployment -ResourceGroupName linux -TemplateFile $template -TemplateParameterFile $param
Write-Host -ForegroundColor Green "Deployed. Pausing for 60 seconds..."
Start-Sleep -Seconds 60
$vm_ip=(Get-AzPublicIpAddress -Name "##################").IpAddress
Write-Host -ForegroundColor Green "`nInitiating SSH connection...`n"

ssh ################@$vm_ip

Bash

project=$1
token="############################"
proj_file="${project}.7z"
root_path="/home/brian/main_projects"

azcopy copy "########################" "/home/brian/main_projects/"
cd $root_path
7z x -y ${root_pth}/${proj_file}

mkdir -p ${root_path}/render/commercial/${project}/

Render -rd "${root_path}/render/commercial/${project}/" ${project}.ma 

azcopy copy "${root_path}/render/commercial/${project}/" "#############################" --recursive --overwrite=ifSourceNewer

echo -e "\e[32mDone! Shutting down in 60 seconds...\n\e[0m"

sudo shutdown
Edited by BrianHanke - Aug. 15, 2022 23:21:54
Subscribe to my Patreon for the best CG tips, tricks and tutorials! https://patreon.com/bhgc [patreon.com]

Twitter: https://twitter.com/brianhanke [twitter.com]
Behance: https://www.behance.net/brianhanke/projects [www.behance.net]
User Avatar
Member
129 posts
Joined: Oct. 2020
Offline
awesome man! wish I can understand a word lol
https://www.youtube.com/channel/UC4NQi8wpYUbR9wLolfHrZVA [www.youtube.com]
User Avatar
Member
160 posts
Joined: Jan. 2019
Offline
Thanks for sharing, I've been following your posts for a while

I'm looking for a way to find all dependences for a Usd file, sublayers, references, textures, vdbs etc. Does anyone know if there is a way of doing this?
User Avatar
Member
433 posts
Joined: April 2018
Offline
I'm interested in figuring that out as well. My scripts aren't too robust in that they require everything to be set up in advance. It assumes that all project geo is in /geo, etc, and doesn't scan for actual paths. That should be possible with any text-based file format (.ma, .usda), but I don't know enough about scripting to say how it would work exactly.
Subscribe to my Patreon for the best CG tips, tricks and tutorials! https://patreon.com/bhgc [patreon.com]

Twitter: https://twitter.com/brianhanke [twitter.com]
Behance: https://www.behance.net/brianhanke/projects [www.behance.net]
User Avatar
Member
273 posts
Joined: Nov. 2013
Online
daviddeacon
I'm looking for a way to find all dependences for a Usd file, sublayers, references, textures, vdbs etc. Does anyone know if there is a way of doing this?

For pure USD you can try https://www.sidefx.com/docs/hdk/dependencies_8h.html#a074224fcf7c0ac30d4a218ba6d90e498 [www.sidefx.com]

from pxr import UsdUtils
dependencies = UsdUtils.ComputeAllDependencies('yourfile.usd')

It should return a tuple of usd layers, assets and unresolved paths.

However if there are other Houdini related dependencies in play that are unknown to USD the above function obviously won't detect those.
User Avatar
Member
160 posts
Joined: Jan. 2019
Offline
antc
For pure USD you can try https://www.sidefx.com/docs/hdk/dependencies_8h.html#a074224fcf7c0ac30d4a218ba6d90e498 [www.sidefx.com]

from pxr import UsdUtils
dependencies = UsdUtils.ComputeAllDependencies('yourfile.usd')

It should return a tuple of usd layers, assets and unresolved paths.

However if there are other Houdini related dependencies in play that are unknown to USD the above function obviously won't detect those.

Thanks for the code snippet, its working great for getting all the Usd file dependencies, I don't suppose you know of a way to find all texture dependencies with in a Usd file?

Thanks again,
David.
User Avatar
Member
273 posts
Joined: Nov. 2013
Online
daviddeacon
Thanks for the code snippet, its working great for getting all the Usd file dependencies, I don't suppose you know of a way to find all texture dependencies with in a Usd file?

Texture paths should be listed in the second element of the returned tuple so long as they are encoded as asset paths in the USD file and not strings. I tried with the alab scene and it seems to be giving me all the textures.
User Avatar
Member
160 posts
Joined: Jan. 2019
Offline
antc
Texture paths should be listed in the second element of the returned tuple so long as they are encoded as asset paths in the USD file and not strings. I tried with the alab scene and it seems to be giving me all the textures.

Thanks, I had totally missed the fact there was a second element in the returned tuple.
  • Quick Links