1) how this line (internally) works? or what is the mathematical concept behind this?
You can see the source code for tf::Pose here. Unfortunately, it's not the most readable code. As for the math, I've illustrated the concept here:
Say you have two poses, pose1 (red) and pose2 (blue). pose1 and pose2 are drawn as vectors from the origin. You want to find the pose diff (green), which is the difference from pose2 to pose1. To do this, you have to "subtract" pose2 from pose1. The mathematical concept here is called homogeneous transformation, part of linear algebra. Each pose can be represented as a 4x4 transformation matrix that contains both translation and rotation (these can also be represented as a vector + a quaternion, but the matrix lets you put both in one object). To compose (or "add") the poses, you multiply the matrices, which is what the * operator does when you "multiply" one pose by another. In this case, because we want to "subtract" pose2 from pose1, we use the inverse, which is a matrix inverse. Also, note that the order of the multiplication matters with matrices--matrix multiplication is not commutative like scalar multiplication is.
2) does this line also calculate the difference between the two quaternions of the poses?
Yes. I wasn't able to draw the orientations into the sketch above, but composition of poses includes rotation as well.
3) Another question is what is the difference between tf::Pose and tf::Transform
Just semantics. On line 50 of geometry/transform_datatypes.h, you'll see they're actually the same type! tf::Pose is just a typedef of tf::Transform. A tf::Pose represents a position+orientation of an object, and it's meant to be a direct analog to the geometry_msgs::Pose message, whereas a tf::Transform is an object that can perform a translation+rotation transform. Mathematically (and in code) they're identical.
