What system would you use to sim and break trees? ATM I'm thinking about rbd for the trunk and vellum for the rest. But for some reason vellum can't holt stiffness of the branches with neither teds or struts. maybe too thin? No setting int eh world makes them stronger.

I'd do everything rbd, but I never done trees tbh

Trying increasing the vellum constraint iterations and solver substeps and see if that improves the stiffness. It will be much slower but might get the result you want.

Probably just a vellum sim on the bones.