Have you built hogman_minimal itself? The libraries should then be in /opt/ros/diamondback/stacks/hogman_minimal/lib (/usr/bin/ld is not a directory but the linker executable). My guess is, that you used the superuser to put hogman_minimal into /opt/ros/... but call make as regular user. Therefore rosmake lacks permission to write the built binaries into the target directory and cannot find them lateron when linking rgbdslam.