I seem to have extremely similar (almost identical) symptoms to an old closed question that got closed with no feedback (https://answers.ros.org/question/262755/nosetests-not-working-reliably-with-catkin_make/). Hopefully this thread doesn't suffer the same fate!

It's a relatively simple python package with some nosetests. The tests fail sometimes and pass sometimes. It seems to be directly correlated with clean build vs rebuild. Clean always fails, non-clean rebuild always passes.

I think it may be a bug in ROS/catkin where the python path is getting set inconsistently on build/rebuild. Here's what I've documented so far:

Scenario A: clean build. Tests fail.

• run rm -r build devel
• run catkin_make
• run source devel/setup.bash
• run catkin_make run_tests or catkin_make run_tests_<package-name>
• Result: errors on including python files from the module this package wraps, e.g. ImportError: No module named <module>
• Without altering any files that would cause a re-build, re-running the same run_tests command produces the same results
• (similar errors for any attempts to include from nested sub-modules)

Scenario B: a dirty re-build, no changes to the files besides last-modified date. Tests pass.

• Setup: Do Scenario A, or just don't wipe build & devel after doing whatever else.
• touch the CMakeLists.txt file in the project in question
• run catkin_make run_tests_<package-name>
• Result: The 'broken' tests files now load & pass successfully.

Additionally, the tests pass if I run them manually by calling nosetests on the command line. My actual PYTHONPATH after sourcing setup.bash is what I would expect (/home/evan//devel/lib/python2.7/dist-packages:/opt/ros/kinetic/lib/python2.7/dist-packages)

I did a little work to verify python path during test execution and here's the steps+results:

• Added the following to the top of my test file: sys.stderr.write('PYTHONPATH=' + os.environ['PYTHONPATH'] + '\n')
• Re-ran Scenario A
  • The debug/stderr output was PYTHONPATH=/opt/ros/kinetic/lib/python2.7/dist-packages
• Re-ran Scenario B
  • The debug/stderr output was PYTHONPATH=/home/evan/<repo-name>/devel/lib/python2.7/dist-packages:/opt/ros/kinetic/lib/python2.7/dist-packages

PYTHONPATH seems like something that should be the same after building & running setup.bash no matter how (nor how many times) I build/run it, not something that changes on a re-build. Is this in fact a ROS/catkin bug of some kind as it seems from the above, or have I somehow misconfigured something?

my setup.py is straightforward/simple:

from distutils.core import setup
from catkin_pkg.python_setup import generate_distutils_setup

# fetch values from package.xml
setup_args = generate_distutils_setup(
    packages=['packagename'],
    scripts=['scripts'],
    package_dir={'': 'scripts'}
)

setup(**setup_args)

My directory structure is

packagename/
  scripts/
    packagename/
      class1.py
      class2.py
      etc.py
      submodule/
        class3.py
  tests/
    class1_test.py
    class2_test.py
    etc_test.py
  CMakeLists.txt
  package.xml
  setup.py

sample test file structure:

import unittest
from mock import Mock
from mock import MagicMock

import os
from packagename.class1 import blah
from packagename.submodule.class3 import blah3

package.xml:

<?xml version="1.0"?>
<package>
  <name>packagename</name>
  <version>0.0.0</version>
  <description>The packagename package</description>
  <maintainer email="blah">blah</maintainer>
  <license>Private</license>

  <buildtool_depend>catkin</buildtool_depend>
  <build_depend>rospy</build_depend>
  <build_depend>msg</build_depend>
  <run_depend>rospy</run_depend>
  <run_depend>msg</run_depend>

  <export>
  </export>
</package>

CMakeLists.txt:

cmake_minimum_required(VERSION 2.8.3)
project(packagename)

find_package(catkin REQUIRED COMPONENTS
  rospy
)

catkin_package(
  INCLUDE_DIRS include
#  LIBRARIES 
  CATKIN_DEPENDS rospy
  DEPENDS msg
)

catkin_python_setup()

include_directories(
  include
  ${catkin_INCLUDE_DIRS}
)

if (CATKIN_ENABLE_TESTING)
  find_package(rostest REQUIRED)

  catkin_add_nosetests(tests DEPENDENCIES msg ${${PROJECT_NAME}_EXPORTED_TARGETS})

endif()

I'm running ROS Kinetic on ubuntu, and I just upgraded to what apt says is "ros-kinetic-desktop-full v1.3.2" today (was hoping maybe it was an old bug that had been fixed in a recent version).

Also, we have a 'msg' package that contains some shared msg and srv definitions (and nothing else). A third package's tests also fail when it tries to import msg.srv. As far as I know, the code all works fine at runtime and can interact with other nodes using those same srv's and msg's. It is only failing during the execution of catkinmake runtests, presumably due to PYTHONPATH not being set correctly. (I tested the pythonpath for those files too and it's also different on clean vs dirty builds)

the msg package's CMakeLists.txt:

cmake_minimum_required(VERSION 2.8.3)

project(msg)

find_package(catkin REQUIRED COMPONENTS
  message_generation
  std_msgs
  sensor_msgs
  genmsg
  roscpp
  rospy
)

add_message_files(
  DIRECTORY msg
  FILES
  Message1.msg
  Message2.msg
  Message3.msg
)

add_service_files(
  DIRECTORY srv
  FILES
  Service1.srv
  Service2.srv
)

generate_messages(
  DEPENDENCIES
  std_msgs
  sensor_msgs
)

catkin_package(
#  INCLUDE_DIRS include
#  LIBRARIES msg
  CATKIN_DEPENDS message_runtime std_msgs sensor_msgs
)

include_directories(
  ${catkin_INCLUDE_DIRS}
)

install(DIRECTORY include/${PROJECT_NAME}/
  DESTINATION ${CATKIN_PACKAGE_INCLUDE_DESTINATION}
  FILES_MATCHING PATTERN "*.h"
)

and package.xml:

<?xml version="1.0"?>
<package>
  <name>msg</name>
  <version>0.0.0</version>
  <description>
    The msg package defines different messages,
    actions and services specific to our applications.
  </description>
  <maintainer email="email@email.com">owner</maintainer>
  <license>Private</license>

  <buildtool_depend>catkin</buildtool_depend>
  <build_depend>message_generation</build_depend>
  <build_depend>std_msgs</build_depend>
  <build_depend>sensor_msgs</build_depend>

  <run_depend>message_runtime</run_depend>
  <run_depend>std_msgs</run_depend>
  <run_depend>sensor_msgs</run_depend>

  <export>
  </export>
</package>

Edits: Corrected setup.py above. Added info about 2nd set of similar errors around the msg project.

Thanks, Evan