Calculating pose in terms of Euler angles for a plane using direction vectors.

edit

Couple of thoughts:

• I think you might want to be asking why you even want the Euler angles. Given the three orthonormal vectors you already have a unique representation of the orientation of that plane in terms of an SO(3) rotation matrix (the three vectors can be thought of as the columns of the rotation matrix). Euler angles have issues with ambiguity (there are 24 different 3-parameter representations of orientation... which one do you mean when you say "Euler angles"), and every one of the 24 representations has singularities. If you can instead do whatever you are trying to do directly with SO(3) or Quaternions, I think your life will actually be a lot easier.
• If you do need to convert the three vectors into some notion of Euler angle, rather than reinventing the wheel, I'd use a library that already supports this. For example in the tf2::Matrix3x3 class there is both getEulerYPR and getRPY.
• I'd double check that the vectors you are computing (however you are getting them) are staying consistent relative to the plane. Many algorithms I've used in the past for plane segmentation have ambiguities and can sometimes return, for example, either the normal vector Z or the inverse of the normal vector. If the vectors are switching signs it's possible that's the root cause of your issue.
• Also, FWIW, I think the equations you list don't actually work for compound rotations. Your expressions are much simpler than what is typically used. See for example, this PDF that lists many of the conversions