ofMatrix4x4 as a return type crashing application

I am using ofxCV, specifically the function
ofMatrix4x4 estimateAffine3D(std::vector<glm::vec3>& from, std::vector<glm::vec3>& to, float accuracy = .99);

I am referencing an example from ofxCV - example-estimate-affine
(https://github.com/kylemcdonald/ofxCv/tree/master/example-estimate-affine). The example works perfectly.

The function above returns an ofMatrix4x4 I am going a bit crazy trying to understand why my program crashes. The following code works perfectly:

ofMatrix4x4 ofApp::findHomographyFromPoints(std::vector<glm::vec3> VRPoints, std::vector<glm::vec3> arucoPoints)
{
ofMatrix4x4 rigidEstimate;
rigidEstimate = estimateAffine3D(VRPoints, arucoPoints);
return rigidEstimate;
}

called like this:
cout << ofToString(findHomographyFromPoints(srcPoints, dstPoints)) << endl;

but if I try to do anything with the returning matrix, like this:

ofMatrix4x4 myMat;
myMat = findHomographyFromPoints(srcPoints, dstPoints);

The program crashes.

I did a little more testing and I also get strange results calling ofMatrix4x4 getInverse() (which I need to do as well)

To make things simpler I make a simple vector of 5 glm::vec3 points and fill them with logical values.

calling
cout << ofToString(findHomographyFromPoints(srcPoints, dstPoints)) << endl;

with the sets of points I see this in the console:

                     1,        0,        0,        0
                     0,        0,        0,        0
                     0,        0,        1,        0
-1.29237e-14,        0,        0,        1

This is as expected, but if I add any variation of getting the inverse of the matrix

ofMatrix4x4 ofApp::findHomographyFromPoints(std::vector<glm::vec3> VRPoints, std::vector<glm::vec3> arucoPoints)
{
ofMatrix4x4 rigidEstimate;
rigidEstimate = estimateAffine3D(VRPoints, arucoPoints).getInverse();
return rigidEstimate;
}

or

ofMatrix4x4 ofApp::findHomographyFromPoints(std::vector<glm::vec3> VRPoints, std::vector<glm::vec3> arucoPoints)
{
ofMatrix4x4 rigidEstimate;
rigidEstimate = estimateAffine3D(VRPoints, arucoPoints);
rigidEstimate = rigidEstimate.getInverse();
return rigidEstimate;
}

With the same sets of points I get:

-nan(ind), -nan(ind), -nan(ind),        0
-nan(ind), -nan(ind), -nan(ind),        0
-nan(ind), -nan(ind), -nan(ind),        0
                  nan,      nan,      nan,        1

In summary the function seems to work, it does not spit out any errors, but it seems that when I try to do anything with the resulting matrix I ether get invalid results or a crash. I am guessing I am doing something pretty dumb that I cannot see.