Since the geometry has no normals, the viewport and Mantra have to automatically compute normals. In order to handle geometry that has clear cusps, e.g. the edges of a cube, it defaults to computing vertex normals with a cusp angle of 60 degrees, so if two adjacent polygons have geometric normals that differ by more than 60 degrees, there will be a cusped edge between them.
If you want to explicitly compute N on the geometry, you can use a Normal SOP with “Add Normals to” set to “Points”. Alternatively, in the viewport display options, under Geometry > Normals > Auto Generate, you can select Point Normals. For Mantra, on the object, you can add a rendering parameter “Add Normals To Geometry” (vm_addnormalsto) and select “Point Normals”. If you want vertex normals, but just a larger cusp angle, there are ways to specify that too. I think there's some way to do this all with detail attributes too, but I can't remember what they're called right now.
Another thing that would probably also make this less of an issue would be if you had higher resolution cloth geometry, since then the angles between adjacent polygons would be less, but that might slow things down, so I understand why that wouldn't be an ideal solution.
Hopefully something above is what you were looking for.
Writing code for fun and profit since... 2005? Wow, I'm getting old.
https://www.youtube.com/channel/UC_HFmdvpe9U2G3OMNViKMEQ [
www.youtube.com]