Turns out the problem was with calling setup.py myself. The documentation explains issues with calling setup.py manually screws things up.
Instead, just uncomment catkin_python_setup() in CMakeLists.txt and follow the instructions in this tutorial.