How is the transformation build up?
Most of the time the map->odom transform is build by SLAM or pure Localization. This is most likely the cause for your movement.
Why should both frames even be fixed? This would imply that you 100% trust your odometry source. This is almost never the case other than in perfect simulation, since you will have things like wheel slip on your robot that will have an impact on your odometry.
That said, if you trust your odometry then there is either no need for the map or for the odom frame you can just publish your map on odom or your odom data straight to map.
This answers is purely based on guesses on your system. If you want precise information you'll need to supply more information about it, maybe a picture of your tf_tree and a tf_graph so we know what exactly is going on.
Btw.
The odometry frame is a continuous frame, where as the map frame is a discrete frame, meaning the transform between map and odom gets calculated in increments and the transform will jump often. As with most questions about TF i recommend REP 105 this is a strict definition how the tf tree is supposed to be set up.