Why can't I import a Python dependency even if it is declared as a runtime dependency in package.xml?

asked 2019-09-22 07:13:50 -0500

nbro
372 ●39 ●47 ●53

updated 2019-09-22 07:38:46 -0500

I have the following dependencies in my package.xml

<buildtool_depend>catkin</buildtool_depend>

<build_depend>message_filters</build_depend>
<build_depend>rosbag</build_depend>
<build_depend>rospy</build_depend>
<build_depend>tf</build_depend>

<build_export_depend>message_filters</build_export_depend>
<build_export_depend>rosbag</build_export_depend>
<build_export_depend>rospy</build_export_depend>
<build_export_depend>tf</build_export_depend>

<exec_depend>message_filters</exec_depend>
<exec_depend>rosbag</exec_depend>
<exec_depend>rospy</exec_depend>
<exec_depend>tf</exec_depend>

<exec_depend>python-h5py</exec_depend>
<exec_depend>python-opencv</exec_depend>
<exec_depend>python-pandas</exec_depend>
<exec_depend>python-tqdm</exec_depend>

I have done rosdep install my_package and catkin build inside my workspace (and I got no error). However, whenever I try to do pip list or to import any of the Python dependencies I specified above, such as Pandas or OpenCV, I get an ImportError or I don't see them in the list. I want to use those Python dependencies in my scripts or source files inside my_package.

If the correct way of specifying Python dependencies is by specifying them in <exec_depend> (see e.g. http://docs.ros.org/lunar/api/catkin/html/howto/format2/python_module_dependencies.html)), then why can't I see pandas when doing pip list?

edit retag flag offensive close merge delete

Comments

What is the output of which python and python --version?

Edit: I doubt <build_export_depend>rospy</build_export_depend> is needed: rospy is a pure Python package. It cannot be a build_export_depend as there are no headers for other (C++) packages to depend on/include.

In general: I would guess 99% of Python dependencies would be only exec_depends.

gvdhoorn ( 2019-09-22 07:23:35 -0500 )edit

The output of the first command is /home/me/catkin_ws/src/my_package/venv/bin/python and of the second is Python 2.7.15+. I am inside a virtual environment, as you can see.

nbro ( 2019-09-22 07:28:24 -0500 )edit

Regarding the <build_export_depend>rospy</build_export_depend>, it was added automatically when I used catkin create pkg --catkin-deps rospy rosbag tf message_filters -- my_package.

nbro ( 2019-09-22 07:31:07 -0500 )edit

Yes, it does that. But that is just because there is no support for Python packages specifically. You can remove all dependencies other than exec_depends from your manifest. Unless you have a mixed C++/Python package, that is.

gvdhoorn ( 2019-09-22 07:37:36 -0500 )edit

@gvdhoorn How do I know which packages contain only Python code? Now, you told me that rospy only needs to be specified as a exec_depend, because it only contains Python code. In general, though, is there a way of knowing whether a dependency only contains Python code, without having to look at the source code?

nbro ( 2019-09-22 07:44:23 -0500 )edit

"only contain Python code" is just a shortcut.

build_export_depend is only used for telling packages that depend on you that they also (transitively) depend on your exported dependencies.

So if you have a header (C++) that includes a.hpp, which comes from package pkg_a, then in order for dependants of your package to be able to include your headers, they must also have pkg_a, or your headers will fail compilation with a cannot find 'pkg_a/a.hpp'.

Python is not a compiled language, so this cannot happen. Ergo: no build_export_depends.

Unless you have a mixed C++/Python package, have C++/C bindings or are doing other more complex things.

gvdhoorn ( 2019-09-22 07:47:37 -0500 )edit

@gvdhoorn However, you did not answer my specific question above about "how do I determine whether a package only needs to be specified as exec_depend, only contains Python code, or is or not required at build time. Maybe I will ask another question on the website.

nbro ( 2019-09-22 08:05:06 -0500 )edit

You really need me to explain you how to determine that a package "only contains Python code"?

There is no single command or fool proof way to see that from anything other than by looking at all the sources that make up the package.

Also: you add build_export_depends to your package. As you are -- presumably -- the author of your own source code, you should know whether you are #include-ing anything from your dependencies in your own headers. You should also know whether you are creating a Python-only package.

build_export_depends communicate to your users which packages your package is using. Not the other way around.

gvdhoorn ( 2019-09-22 08:07:24 -0500 )edit

see more comments

answered 2019-09-22 07:36:37 -0500

gvdhoorn
86574 ●283 ●1432 ●1054 http://cor.tudelft.nl/

updated 2019-09-22 08:29:48 -0500

The output of the first command is /home/me/catkin_ws/src/my_package/venv/bin/python and of the second is Python 2.7.15+. I am inside a virtual environment, as you can see.

This is most likely the cause of you not being able to import the packages listed in the package manifest: rosdep will use your OS' package manager -- which in the case of Ubuntu (and Debian) would be apt -- to install the system dependencies that you've listed in your package's manifest.

For python-pandas for instance, it would lookup the rosdep rule for that key (here) and resolve it to python-pandas (in this case the rosdep key resolves to the same Ubuntu/Debian package name, but that is a coincidence).

It would then ask apt to install the python-pandas Debian package. On Ubuntu (and Debian), that would result in the Python 2 version of that package getting installed in the default system location: /usr/lib/python2.7/dist-packages/pandas. There is also a Python 3 version of the pandas rosdep key (here), but that wouldn't change what happens (it would just install the Python 3 version of Pandas in the system default location).

apt does not take any virtual environments into account, it will just install in the system default location (or really: it installs files in whichever location is specified in the .deb package).

This immediately means that the packages are installed, so rosdep will be happy (as apt reports it has installed whatever rosdep asked of it), but the packages will not be available in your virtual environment.

However, whenever I try to do pip list [..] I don't see them in the list.

this should be clear now: pip will look in your virtual environment, and as the packages have not been installed there, it will not list them as available.

Edit:

is there a way of installing the dependencies in the virtual environment with a ROS tool or I need to e.g. install the dependencies manually (from e.g. a requirements.txt file)?

rosdep will only use apt on Ubuntu/Debian and pip for certain dependencies (typically recognisable by the -pip suffix on their rosdep key). As Python virtualen environments don't wrap/redirect apt, nor where apt installs Python modules, I would classify this as a shortcoming of how Python virtual environments work (combined with how rosdep prefers apt on Ubuntu/Debian and how Ubuntu/Debian prefers to package Python modules as .deb packages).

There is no way I know of that would let you list just the Python dependencies of a ROS package. You could perhaps use rospack rosdep $pkg_name to get a list of system dependencies (which Python pkgs typically are), somehow filter those to leave just the Python dependencies, and feed that list to the pip in your virtual environment.

I've never done something like that, so I have no idea how well that would work.

Providing a requirements.txt to your users (which would ... (more)

Comments

The same 'problem' can arise with any Python interpreter other than the system default one btw. It's not unique to virtual environments.

gvdhoorn ( 2019-09-22 07:39:47 -0500 )edit

Yes, you are apparently right. If I deactivate the virtual environment, I can now see the Python dependencies and import them. However, what if I did not want to pollute my global Python distribution while developing, is there a way of installing the dependencies in the virtual environment with a ROS tool or I need to e.g. install the dependencies manually (from e.g. a requirements.txt file)?

nbro ( 2019-09-22 08:19:51 -0500 )edit

just remembered there is actually a way to have rosdep output rules only for specific installers. You can pass it the --filter-for-installers option with as value pip for instance. This only works with the db command though.

gvdhoorn ( 2019-09-22 08:34:20 -0500 )edit

add a comment

Why can't I import a Python dependency even if it is declared as a runtime dependency in package.xml?

Comments

1 Answer

Comments

Question Tools

Stats

Related questions

Why can't I import a Python dependency even if it is declared as a runtime dependency in package.xml? edit

Comments

1 Answer

Comments

Question Tools

Stats

Related questions

Why can't I import a Python dependency even if it is declared as a runtime dependency in package.xml?