You'll want to declare dependencies on both packages in your package.xml, and then use the find_package() command in CMake to locate them and add the appropriate include directories and libraries to your project.
There's a fairly good catkin build on how to declare and use external dependencies: http://docs.ros.org/hydro/api/catkin/...
It looks like GTSAM is already a CMake package, so you should be able to use find_package with it without issues. I did a quick google search and found someone at Georgia Tech doing this from rosbuild: https://code.google.com/p/gt-ros-pkg/... ; you'll probably need to change the rosbuild macros to normal cmake macros, but otherwise you should be able to do something similar.
Since you're installed GTSAM from source, and most of the other people that use your package will probably do the same thing, you may not want to declare your dependency on gtsam in your package.xml ; this will prevent the ROS tools from trying to verify that it's installed, which is difficult to do for source packages.
If you do get to a point where you'd like to release your package to the ROS build farm, you'll probably want to get the GTSAM maintainers to release it as a plain CMake package into ROS. This is fairly easily done through Bloom.
Side notes on Boost:
- ROS depends on Boost. You may hit issues if you try to compile against a different version of Boost than the rest of ROS.
- If you're on a Mac and have Boost installed through homebrew, you should be fine with the usual procedure. You'll be using the same version of Boost that the rest of ROS.
- If you're on a different system, you'll probably hit issues if you're trying to depend on a different version of Boost than the rest of ROS.
- If you don't need any of the features in the latest version of Boost, I would strongly suggest trying to use the system version of Boost
- If you REALLY need a newer version of Boost, you can try to build all of ROS from source against the newer version to avoid version conflicts.
