So… I've been down a rabbit hole, but I've made it to a proper result!
Sadly enough the rig you provided had the rotations compeletly flip 180 degrees rather randomly. Everything did move though.
Since we're in the case of this rig already following 3 points on the meshes and with the help of a more mathematical cued in brain than mine and the maya API I tried a completely different approach.
Initially I tried following the first 4 vertices to create tetrahedron postion matrixes.
I keep the start state of each matrix and create a new one per frame.
As a note the maya API likes to garbadge collect allot of it's more primitive objects if you leave the scope of a method (this includes class inits). So I had to create a new OpenMaya.MMatrix for the start frame for every frame bake.
I've had to deal with similar conflicts with garbage collection when querrying vert colours on a mesh.
The magical function is:
startPositions . TransformMatrix = newPositions
Since we're looking for the transform matrix we transform the function to:
transformMatrix = inverse(startPositions) * newPositions
in practice this leads to the following lines of code:
transformMatrix = startPositions.inverse() * newPositions
transformMatrix = om.MTransformationMatrix(transformMatrix)
in this om is maya.api.OpenMaya and the position variables are an instance of MMatrix where each row is a point as (x, y, z, 1)
There was one more snag though. Some objects could really go of kilter with this approach. The assumption was that some of the tetrahedrons we were collecting were almost a strip of points. Leading to a situation, as I understand it, that the matrix can't be correctly inversed.
Maya doesn't block against bad inverse input.
So we changed the points to the first 3 points of a polygon and a fourth point in the center of these 3 points offset by this poly it's normal. Just to guarentee a sensible shape being used without too much validation code.
In practice to get the point matrix you run code like this:
polyNormal = self.fnMesh.getPolygonNormal(0)
vertices = self.fnMesh.getPolygonVertices(0)
point0 = self.fnMesh.getPoint(vertices, om.MSpace.kWorld)
point1 = self.fnMesh.getPoint(vertices, om.MSpace.kWorld)
point2 = self.fnMesh.getPoint(vertices, om.MSpace.kWorld)
x_center = (point0.x + point1.x + point2.x) /3
y_center = (point0.y + point1.y + point2.y) /3
z_center = (point0.z + point1.z + point2.z) /3
point3 = om.MPoint(polyNormal.x + x_center, polyNormal.y + y_center, polyNormal.z + z_center)
values = [point0, point1, point2, point3]
values = [(p, p, p, p) for p in values]
tetraMatrix = om.MMatrix((values, values, values, values))
Technically this approach will support meshes that translate, rotate and scale.
I'm sharing the info in case anybody else googles their way to this thread and to confirm that my issues for at least roundabout animation baking houdini destruction are resolved.
Thx for all the input and help allong the way