Ask Your Question
0

Running a prerelease test fails with Python ImportError

asked 2019-08-02 03:28:31 -0600

MarcoStb gravatar image

updated 2019-08-02 04:33:58 -0600

Hey everyone,

I am currently trying on publishing a ROS package and following the Realeasing A Package guide. So far I've added my package name to the rosdistro for kinetic and verified its readiness before. Unfortunately, I can't manage to execute a prerelease test. I get stuck at the 4th step of section 5.2. When I run the generated command (which is shown below), I get the following error:

The command:

mkdir -p /tmp/prerelease_job
cd /tmp/prerelease_job
generate_prerelease_script.py \
  https://raw.githubusercontent.com/ros-infrastructure/ros_buildfarm_config/production/index.yaml \
  kinetic default ubuntu xenial amd64 \
  robot_statemachine \
  --level 1 \
  --output-dir ./

The error output:

Traceback (most recent call last):
  File "/usr/bin/generate_prerelease_script.py", line 35, in <module>
    from ros_buildfarm.config import get_index as get_config_index
  File "/usr/lib/python3/dist-packages/ros_buildfarm/config/__init__.py", line 20, in <module>
    import yaml
  File "/usr/lib/python2.7/dist-packages/yaml/__init__.py", line 2, in <module>
    from error import *
ImportError: No module named 'error'

So far, I've tried with the first suggestion under 5.1, installing the python3-ros-buildfarm which produced the above error. Then I removed it, and installed python-ros-buildfarm, also installing the dependencies required in the second purple box under section 5.1. With this the prerelease script could be built but it produced the following error when I ran it:

prerelease.sh output with python-ros-buildfarm:

Prerelease script

By default this script will continue even if tests fail.
If you want the script to abort and return a non-zero return code
you can set the environment variable ABORT_ON_TEST_FAILURE=1.
You can also set ABORT_ON_TEST_FAILURE_UNDERLAY=1 or
ABORT_ON_TEST_FAILURE_OVERLAY=1 to only affect a specific workspace.

Use workspace: /tmp/prerelease_job


Clone source repositories for underlay workspace

+ git clone --recurse-submodules -b master https://github.com/MarcoStb1993/robot_statemachine.git ws/src/robot_statemachine
Cloning into 'ws/src/robot_statemachine'...
remote: Enumerating objects: 63, done.
remote: Counting objects: 100% (63/63), done.
remote: Compressing objects: 100% (41/41), done.
remote: Total 936 (delta 31), reused 37 (delta 19), pack-reused 873
Receiving objects: 100% (936/936), 1.90 MiB | 587.00 KiB/s, done.
Resolving deltas: 100% (627/627), done.
Checking connectivity... done.
+ git -C ws/src/robot_statemachine --no-pager log -n 1
commit 6217fdb14c265dbf85b4688b730f8514caf30efa
Author: MarcoStb1993 <35468630+MarcoStb1993@users.noreply.github.com>
Date:   Thu Aug 1 12:05:33 2019 +0200

    Delete .project



Clone release repositories for overlay workspace

Underlay workspace contains 6 packages:
- robot_statemachine
- statemachine
- statemachine_additions
- statemachine_msgs
- statemachine_rqt_plugins
- statemachine_rviz_plugins
Overlay packages based on dependency level: 0
Overlay packages based on whitelisted package names: 0
Overlay workspace will contain 0 packages:

Traceback (most recent call last):
  File "/usr/bin/vcs", line 11, in <module>
    load_entry_point('vcstool==0.2.2', 'console_scripts', 'vcs')()
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 542, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2569, in load_entry_point
    return ep.load()
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2229, in load
    return self.resolve()
  File "/usr/lib/python2.7/dist-packages/pkg_resources ...
(more)
edit retag flag offensive close merge delete

Comments

off-topic, but: is your package really called statemachine? That is a really generic name and I'm surprised your ros/rosdistro#21873 was merged.

I can only make the suggestion that you rename your packages to something more descriptive and unique. Refer to REP 144: ROS Package Naming for some rationale on why using descriptive names is much preferred over using generic ones.

gvdhoorn gravatar imagegvdhoorn ( 2019-08-02 04:27:39 -0600 )edit

As to the error message: you appear to be mixing Python 2.7 and Python 3 modules and interpreters.

What is the output of which python, which python3 and python --version?

gvdhoorn gravatar imagegvdhoorn ( 2019-08-02 04:28:29 -0600 )edit

Hm, yes, now that you say it ... It is bundled in a metapackage called robot_statemachine though, if that makes it any better.

MarcoStb gravatar imageMarcoStb ( 2019-08-02 04:29:50 -0600 )edit

I supsected this to be the case, but I was not able to fix it on my own. Here are the outputs:

  • which python: /usr/bin/python
  • which python3: /usr/bin/python3
  • python --version: Python 2.7.12
  • python3 --version: Python 3.5.2

Added them to the post as well.

MarcoStb gravatar imageMarcoStb ( 2019-08-02 04:32:05 -0600 )edit

Hm, yes, now that you say it ... It is bundled in a metapackage called robot_statemachine though, if that makes it any better.

Not really.

Metapackages don't "bundle" other packages. They merely provide convenience.

I've written something about them in #q256493.

gvdhoorn gravatar imagegvdhoorn ( 2019-08-02 04:32:53 -0600 )edit

1 Answer

Sort by ยป oldest newest most voted
1

answered 2019-08-02 04:38:44 -0600

gvdhoorn gravatar image

updated 2019-08-03 05:44:46 -0600

Here are the outputs:

  • which python: /usr/bin/python
  • python --version: Python 2.7.12

The default Python interpreter appears to be Python 2.7 on your system. The Python scripts in the ros_buildfarm package only specify python as the required interpreter (here fi), so on your system that will result in Python2 being used. This can lead to the problems that you experienced.

I believe you should either execute the script as python3 generate_prerelease_script.py .. or create a virtualenv with only Python 3 enabled and use that to run the scripts and tests.

Personally I would recommend the virtualenv approach.

I can't access the wiki page you link to right now (host appears to be busy with other things), but at least the ros_buildfarm Setup environment to deploy configuration document suggests using a virtualenv as well.


Edit: the following steps work for me to create a virtualenv with all dependencies installed:

virtualenv -p python3 $HOME/venv_ros_buildfarm
source $HOME/venv_ros_buildfarm/bin/activate
pip3 install empy
pip3 install jenkinsapi
pip3 install rosdistro
pip3 install ros_buildfarm

At this point all the scripts in ros_buildfarm should be available and all use the python3 interpreter.

You'll have to activate this virtualenv whenever you want to run the prerelease test(s) as well I believe.


Edit 2: venv setup steps were updated in ros-infrastructure/ros_buildfarm#658.

edit flag offensive delete link more

Comments

Thanks for the response, but unfortunately I couldn't manage to get virtualenv running. I followed the approach in Setup environment to deploy configuration but at the command curl https://bootstrap.pypa.io/get-pip.py | python3 I got the following error:

File "/tmp/tmphgxcgthz/pip.zip/pip/_internal/wheel.py", line 7, in <module>
  File "/usr/lib/python3.5/compileall.py", line 20, in <module>
    from concurrent.futures import ProcessPoolExecutor
  File "/usr/lib/python2.7/dist-packages/concurrent/futures/__init__.py", line 8, in <module>
    from concurrent.futures._base import (FIRST_COMPLETED,
  File "/usr/lib/python2.7/dist-packages/concurrent/futures/_base.py", line 357
    raise type(self._exception), self._exception, self._traceback
                               ^
SyntaxError: invalid syntax
MarcoStb gravatar imageMarcoStb ( 2019-08-02 05:48:45 -0600 )edit

I don't know how to run the first approach. When I try it, I get the following output: python3: can't open file 'generate_prerelease_script.py': [Errno 2] No such file or directory.

MarcoStb gravatar imageMarcoStb ( 2019-08-02 05:50:17 -0600 )edit

pip is typically already installed in a virtualenv. I would drop the --without-pip when creating the virtualenv.

And you obviously also don't need to use Python 3.4, just use whatever Python version you have.

Btw: it would seem even this command mixes Python 2 and 3 modules. Are you automatically sourceing (multiple) workspaces in your .bashrc or something similar?

gvdhoorn gravatar imagegvdhoorn ( 2019-08-02 05:51:34 -0600 )edit

I don't know how to run the first approach. When I try it, I get the following output: python3: can't open file 'generate_prerelease_script.py': [Errno 2] No such file or directory.

that's because you probably don't have the Python 3 version installed any more:

Then I removed it

In any case: I would use a virtualenv.

gvdhoorn gravatar imagegvdhoorn ( 2019-08-02 05:52:34 -0600 )edit

I've added the steps that work for me, but I have the impression that your Python setup may be in a bit of a weird state, seeing the error messages you show mix Python 2.7 and Python 3 paths.

gvdhoorn gravatar imagegvdhoorn ( 2019-08-02 06:06:10 -0600 )edit

Thanks a lot for your help! Without the --without-pip and using only pyvenv venv it seems to be working (still building stuff, but that's probably a good sign). I know my Python setup seems to be really messed up, I'll try to clean it up soon. Also, you were right about the .bashrc, I was sourcing the ros and my local catkin workspace in there. I've commented those out. Do you know if it is possible to setup the virtualenv so that I don't have to install empy, jenkinsapi, rosdistro and ros_buildfarm every time anew?

MarcoStb gravatar imageMarcoStb ( 2019-08-02 06:22:43 -0600 )edit

Do you know if it is possible to setup the virtualenv so that I don't have to install empy, jenkinsapi, rosdistro and ros_buildfarm every time anew?

that's not something that I recognise and should not be necessary. Unless you create the virtualenv in your /tmp of course.

Are you confused by this statement:

You'll have to activate this virtualenv whenever you want to run the prerelease test(s) as well I believe.

activate != recreate. The virtualenv will persist. You just need to source /path/to/your/virtualenv/bin/activate every time you want to use it.

gvdhoorn gravatar imagegvdhoorn ( 2019-08-02 06:25:14 -0600 )edit

Thanks again for clarifying and helping me out so much! I never used virtualenv before, so wasn't quite sure what I was doing. o I found out that the scripts are not working as soon as I have sourced /opt/ros/kinetic/setup.bash or /home/user/catkin_ws/devel/setup.bash. If I open a new terminal, the commands will be executed without an issue, even without a virtualenv. I guess this probably isn't normal behaviours ...

MarcoStb gravatar imageMarcoStb ( 2019-08-03 07:24:08 -0600 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

1 follower

Stats

Asked: 2019-08-02 03:28:31 -0600

Seen: 39 times

Last updated: Aug 03