I've got a female figure that I've set up for FEM simulation so that her breasts move naturally. She's wearing a sports bra that I want to compress her breasts and restrict their motion. I have yet to figure out how to actually achieve that. I'm still fairly new at FEM, and also Vellum simulations at DOP rather than SOP level (which so far doesn't seem to work as well as SOP Vellum sims). The figure is set up in a FEM solid object that feeds a FEM solver, and that works just great on its own. The Vellum objects (I'm also simulating her pants in the DOP while I'm at it) are merged as mutual colliders and connected to a Vellum solver, which works on its own. The problem is that when I merge the two together, the behavior gets a little wonky. The merge is set to mutual collider, but when the FEM solver is the first input, the bra ends up being pushed above the breasts by the end of the simulation; with the FEM solver as the second input, the FEM and Vellum simulations just kind of...ignore each other. The FEM simulation starts walking, but the bra simulates as if it were stationary. I've also tried setting the merge to "constraint", and I don't remember what effect that had, but it didn't solve the problem. What am I missing?

To achieve the best possible results for a two-way interaction like this, I would recommend using only a single solver for the entire setup.
You could use either FEM or Vellum, whatever your preference is.
For the bra, I would recommend creating a tube-shaped polygon mesh around the body.
This tube could have collisions turned off and interact with the body using attach and/or slide constraints, which are available in some form on both solvers.

So merge the Vellum object and the FEM solid object, then feed that merge into one of the solvers?

gordig
So merge the Vellum object and the FEM solid object, then feed that merge into one of the solvers?

each solver must be used only with compatible objects so you can use
Cloth Object + Solid object and FEM Solver
or
Vellum Cloth + Vellum Tetrahedral Softbody and Vellum Solver
I'd personally go with Vellum

You reminded me that I haven't really played around too much with tetrahedral Vellum solutions, so maybe I should give that a go.

definitely try vellum tetrahedral softbody since Muscle tools are also using those and 19.5 got a few updates that were getting the behavior closer to FEM

you can watch this https://vimeo.com/739693945 [vimeo.com] ideally from beginning, but at ~ 9:20 John compares it to FEM

A lot of that video went over my head for now, but definitely something to tuck into later. Thank you.

I found some time to try out Vellum Tet sims, and I'm having three problems with it:

1. I can't figure out how to set the strength high enough that the figure holds its shape well enough for my liking
   
2. Because of #1, I also can't scale the stiffness with attributes such that the only parts that deform are the ones I want to deform.
   
3. Regardless of #1 and #2, I can't figure out how to run the animated figure through the simulation. I'm guessing I'll have to simulate the figure in base pose, then drive the simulated figure through a point deform, much like with a FEM sim.
   

I've got the whole figure fed into a Tet Conform, with a created point attribute of "targetstrength" (carried over from FEM simulations, because I'm more used to those) set to 1e9. The breasts and butt are made out of primitive spheres booled away from the base figure, Tet Embeded and given their own attribute of targetstrength, which is currently set to 0, but neither of those values seem to matter much, whether they're scaling the stretch stiffness or the compression stiffness. It doesn't help that Vellum Tet sims don't have bend stiffness.

gordig
I found some time to try out Vellum Tet sims, and I'm having three problems with it:

1. I can't figure out how to set the strength high enough that the figure holds its shape well enough for my liking
   
2. Because of #1, I also can't scale the stiffness with attributes such that the only parts that deform are the ones I want to deform.
   
3. Regardless of #1 and #2, I can't figure out how to run the animated figure through the simulation. I'm guessing I'll have to simulate the figure in base pose, then drive the simulated figure through a point deform, much like with a FEM sim.
   

1. your figure seems pretty stiff, probably more than you need so I don;t think it needs to be stiffer
2. you don't need to vary the tet stiffness to get what you are after, what you want is to add Soft Pin Constraint with Match Animation checked and let that define what parts follow the animation of target geometry with full stiffness ( areas aroud "skeleton") and which are completely free, or anything inbetween
so if you use your "targetstrength" parameter for that make is 0-1 mask as stiffness scaling parameters are applied logarithmically

3. Pin To Target Constraint with Match Animation on and control the stiffness like mentioned in 2.

to optimize the setup you can also use Hard Pin constraint for points that you want to fully follow animation (like areas around "bones") instead of having those also be very stiff soft pin constraints, however you may run into some issues and jitter if you force hard constrained points into some impossible situations

Thanks to your help, I've been able to hammer this into a workable shape, so all that's left is fine-tuning, which brings me to probably the final question of this thread: is there any logic to how one balances things like stiffness, damping ratio and so on? Some kind of guide or rule of thumb? So far I've just been making incremental changes to different parameters and seeing what changes.