any tips on getting the simulation to calm down without crazy substepping?

Try scaling everything up by 10x, simulate, and then scale it back down..?

In my experience flip is less about the actual scale and more about ratio among scale forces and time (unless your are dealing with scales that start running into rounding errors, usually further from origin)

So let's say you are using realistic units for scale, gravity and other forces

Scaling it 10x would also imply scaling your forces 10x so you would most likely still need the same amount of substeps (If you don't scale your forces then most likely you can get away with much less substeps as the acceleration will generally be 0.1x but obviously very different feel)
Or you can also just scale down your time which will require less substeps as each frame now represents a substep of real time
Not necessarily solving the issue as you will need more frames but a lot of times it's better for slowmo shots as you would have all substeps stored

One way with substeps though is to make sure all your colliders and sources are cached, held or interpolated for all substeps so that no extra work is done on them during the solve and therefore the solve can run much faster