I want to track a specific person using skeleton openni tracker node available. However, this node track multiple people which is not good for my application.

I think about SURF in opencv and got it working to track the pattern on t shirt. But this is in rgb camera frame and how can i utilize this with skeleton tracking.

I also think about making unique pose with skeleton by comparing the left hand to left shoulder joint in every users ID skeleton at the start. if left hand is above the left shoulder, i will track that person. However, how can i lookup that transform for each user? I try to modify the code in openni tracker node but no luck so far since the user can be lost or new user step in, tf frame changes everytime

It would be great if someone help me to align rgb image with depth image