I have several Vellum objects starting with intersections/overlaps, and I want them to separate before the actual simulation begins.

In RBD/Bullet I can use:

i@found_overlap = 1;

to force the pieces to resolve overlaps.

Is there an equivalent workflow or attribute for Vellum? I’m looking for a way to untangle/separate the geometry first, then let the normal Vellum simulation start.

Do you want to move the pieces rigidly to resolve overlaps or can they be deformed?

If you don't mind the pieces being distorted, you can use a Muscle Deintersect SOP. Or, if you want the pieces to remain rigid, I set up a little example file that moves the pieces using the Detangle SOP as a starting point for calculating what direction to move each piece in.

The alternative is to use Bullet for the initial configuration, and then run the Vellum sim.

Hello. As Liesbeth Levick said, the most reliable way would be to actually use @found_overlap in RBD but configure it to cook on the first frame by enabling "Solve on Creation Frame". So, after Copy to points node preserve name attribute by renaming it to @orig_name and deleting @name. RBD sim requires unique per-piece name attribute so generate it in RBD configure sop but under a condition that you haven't previously created a @name attribute. In case of feeding @name into rbd config, it won't override it and bullet solver won't work. Then feed that i@found_overlap = 1; into solver to cook only on first frame. Then clean up by freezing on frame 1 with timeshift, unpack and delete that newly created RBD @name attribute and extract preserved @orig_name into @name.
I appended an example hip file.