How to avoid mismatching borders when tracing height fields with increasing thresholds?

This seems tricky, since it's going to interpolate between the voxels.
You could fuse to a grid after tracing, using the grid spacing of the hf (and maybe a slight smooth after)
Otherwise you'll probably need to cook up a trace that does not interpolate between voxels.

Thank you AslakKS,

I ended up setting the tracer's boundary color to a really low value which seems to avoid this issue.