catkin_add_nosetests runs with inconsistent pythonpath (depending whether it's a clean build or a rebuild)

asked 2018-05-15 19:34:05 -0500

Evan Gridley gravatar image

updated 2018-05-16 13:30:43 -0500

I seem to have extremely similar (almost identical) symptoms to an old closed question that got closed with no feedback ( https://answers.ros.org/question/2627... ). Hopefully this thread doesn't suffer the same fate!

It's a relatively simple python package with some nosetests. The tests fail sometimes and pass sometimes. It seems to be directly correlated with clean build vs rebuild. Clean always fails, non-clean rebuild always passes.

I think it may be a bug in ROS/catkin where the python path is getting set inconsistently on build/rebuild. Here's what I've documented so far:

Scenario A: clean build. Tests fail.

  • run rm -r build devel
  • run catkin_make
  • run source devel/setup.bash
  • run catkin_make run_tests or catkin_make run_tests_<package-name>
  • Result: errors on including python files from the module this package wraps, e.g. ImportError: No module named <module>
  • Without altering any files that would cause a re-build, re-running the same run_tests command produces the same results
  • (similar errors for any attempts to include from nested sub-modules)

Scenario B: a dirty re-build, no changes to the files besides last-modified date. Tests pass.

  • Setup: Do Scenario A, or just don't wipe build & devel after doing whatever else.
  • touch the CMakeLists.txt file in the project in question
  • run catkin_make run_tests_<package-name>
  • Result: The 'broken' tests files now load & pass successfully.

Additionally, the tests pass if I run them manually by calling nosetests on the command line. My actual PYTHONPATH after sourcing setup.bash is what I would expect (/home/evan/<repo-name>/devel/lib/python2.7/dist-packages:/opt/ros/kinetic/lib/python2.7/dist-packages)

I did a little work to verify python path during test execution and here's the steps+results:

  • Added the following to the top of my test file: sys.stderr.write('PYTHONPATH=' + os.environ['PYTHONPATH'] + '\n')
  • Re-ran Scenario A
    • The debug/stderr output was PYTHONPATH=/opt/ros/kinetic/lib/python2.7/dist-packages
  • Re-ran Scenario B
    • The debug/stderr output was PYTHONPATH=/home/evan/<repo-name>/devel/lib/python2.7/dist-packages:/opt/ros/kinetic/lib/python2.7/dist-packages

PYTHONPATH seems like something that should be the same after building & running setup.bash no matter how (nor how many times) I build/run it, not something that changes on a re-build. Is this in fact a ROS/catkin bug of some kind as it seems from the above, or have I somehow misconfigured something?

my setup.py is straightforward/simple:

from distutils.core import setup
from catkin_pkg.python_setup import generate_distutils_setup

# fetch values from package.xml
setup_args = generate_distutils_setup(
    packages=['packagename'],
    scripts=['scripts'],
    package_dir={'': 'scripts'}
)

setup(**setup_args)

My directory structure is

packagename/
  scripts/
    packagename/
      class1.py
      class2.py
      etc.py
      submodule/
        class3.py
  tests/
    class1_test.py
    class2_test.py
    etc_test.py
  CMakeLists.txt
  package.xml
  setup.py

sample test file structure:

import unittest
from mock import Mock
from mock import MagicMock

import os
from packagename.class1 import blah
from packagename.submodule.class3 import blah3

package.xml:

<?xml version="1.0"?>
<package>
  <name>packagename</name>
  <version>0.0.0</version ...
(more)
edit retag flag offensive close merge delete

Comments

Is your setup.py the real one, and is the directory structure you show an example, or the other way around?

Additional note: your CMakeLists.txt contains quite some unnecessary things for a Python pkg. catkin_package() can be empty, and rospy does not need to be find_package()-ed. Also ..

gvdhoorn gravatar image gvdhoorn  ( 2018-05-16 01:43:45 -0500 )edit

.. you don't have any include_directories(..), so remove that.

gvdhoorn gravatar image gvdhoorn  ( 2018-05-16 01:44:07 -0500 )edit

Corrected inline above. Setup.py was the "real" one. (they're all "real", and do properly reference each other in our repo. I just changed the directory/package/file names to something generic but apparently I did it inconsistently) Thanks for the notes.

Evan Gridley gravatar image Evan Gridley  ( 2018-05-16 11:31:26 -0500 )edit

I don't think packagename can be in the scripts directory. At least not if you want it to be seen as a module.

gvdhoorn gravatar image gvdhoorn  ( 2018-05-16 11:49:25 -0500 )edit

Hmm... It works fine after a dirty rebuild as mentioned above. The python path is only wrong during the execution of catkin_make run_tests. Our application using this code has always worked fine.

Also, this failure showed up suddenly after an unrelated commit. Tests all passed on CI 2 days ago.

Evan Gridley gravatar image Evan Gridley  ( 2018-05-16 13:15:44 -0500 )edit

In general I would recommend not to source the setup file in the same shell where that workspace is being built. A repeated build with the setup file sourced will certainly have a different environment than the first build. Does the same problem exist when you don't source the setup file?

Dirk Thomas gravatar image Dirk Thomas  ( 2019-01-02 13:32:21 -0500 )edit