ROS Resources: Documentation | Support | Discussion Forum | Index | Service Status | ros @ Robotics Stack Exchange
Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

answered 2020-09-13 19:55:48 -0500

33thou gravatar image

Answering my own question here:

From FindPythonExtra.cmake I determined the culprit to be Fish shell interacting poorly with my pyenv + pipenv environment.

This was exacerbated by the cmake cache, which persisted incorrectly returned results from python3-config --ldflags between runs.

For future searchers using pyenv:

  • Ensure that PYTHON_CONFIG_EXECUTABLE in FindPythongExtra finds the correct (shimmed) version of python3-config
  • Pass --cmake-clean-cache to colcon build to ensure incorrect values are not persisted between runs

There's also an outstanding pyenv issue (CMake might not find the correct python version) which might be related.

Answering my own question here:

From FindPythonExtra.cmake I determined the culprit to be cmake and possibly Fish shell interacting poorly with my pyenv + pipenv environment.

This was exacerbated by the cmake cache, which persisted incorrectly returned results from python3-config --ldflags between runs.

For future searchers using pyenv:

  • Ensure that PYTHON_CONFIG_EXECUTABLE in FindPythongExtra finds the correct (shimmed) version of python3-config
  • Pass --cmake-clean-cache to colcon build to ensure incorrect values are not persisted between runs

There's also an outstanding pyenv issue (CMake might not find the correct python version) which might be related.

Answering my own question here:

From FindPythonExtra.cmake I determined the culprit to be cmake some kind of interaction between cmake, python3-config, brew, pyenv, and possibly Fish shell interacting poorly with my pyenv + pipenv environment..

This was exacerbated by the cmake cache, which persisted incorrectly returned results from python3-config --ldflags between runs.

For future searchers searchers:

First, Note that for debugging, it's useful to pass --cmake-args=" --log-level=STATUS" --cmake-clean-cache to colcon in order to see status messages and avoid caching broken values of the python3-config output between runs.

It appears that when you're using pyenv:

  • Ensure that PYTHON_CONFIG_EXECUTABLE pyenv+pipenv and your virtual environment is pointing to a brew-installed python, python3-config returns incorrect information. For example:

    (ros2_foxy) ➜  ros2_foxy which python  
/Users/d/ros2_foxy/.venv/bin/python
(ros2_foxy) ➜  ros2_foxy python --version        
Python 3.8.5
(ros2_foxy) ➜  ros2_foxy which python3-config
/Users/d/.pyenv/shims/python3-config
(ros2_foxy) ➜  ros2_foxy python3-config --ldflags
-L/Users/d/.pyenv/versions/3.7.6/lib/python3.7/config-3.7m-darwin -lpython3.7m -ldl -framework CoreFoundation
(ros2_foxy) ➜  ros2_foxy python3-config --includes

    -I/Users/d/.pyenv/versions/3.7.6/include/python3.7m -I/Users/d/.pyenv/versions/3.7.6/include/python3.7m

    This results in FindPythongExtra finds the correct (shimmed) the ros2 _FindPythonExtra.cmake_ file failing with the error Could NOT find PythonExtra (missing: PythonExtra_LIBRARIES).

    On the other hand, if pipenv and pyenv are set up to use a pyenv-installed rather than brew-installed version of Python in the virtual environment, things seem to work. This can be done by explicitly setting an alternate version, e.g. 

    ➜  ros2_foxy pipenv --python 3.7
Creating a virtualenv for this project…
Pipfile: /Users/d/ros2_foxy/Pipfile
Using /Users/d/.pyenv/versions/3.7.6/bin/python3.7m (3.7.6) to create virtualenv…
⠏ Creating virtual environment...created virtual environment CPython3.7.6.final.0-64 in 476ms
  creator CPython3Posix(dest=/Users/d/ros2_foxy/.venv, clear=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/Users/d/Library/Application Support/virtualenv)
    added seed packages: pip==20.2.1, setuptools==49.6.0, wheel==0.35.1
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator

✔ Successfully created virtual environment! 
Virtualenv location: /Users/d/ros2_foxy/.venv
Creating a Pipfile for this project…

    Now the results of python3-config

  • Pass --cmake-clean-cache to are consistent with the virtual environment:

    (ros2_foxy) ➜  ros2_foxy which python
/Users/d/ros2_foxy/.venv/bin/python
(ros2_foxy) ➜  ros2_foxy python --version
Python 3.7.6
(ros2_foxy) ➜  ros2_foxy which python3-config
/Users/d/.pyenv/shims/python3-config
(ros2_foxy) ➜  ros2_foxy python3-config --ldflags
-L/Users/d/.pyenv/versions/3.7.6/lib/python3.7/config-3.7m-darwin -lpython3.7m -ldl -framework CoreFoundation
(ros2_foxy) ➜  ros2_foxy python3-config --includes
-I/Users/d/.pyenv/versions/3.7.6/include/python3.7m -I/Users/d/.pyenv/versions/3.7.6/include/python3.7m

    And rcutils builds:

    (ros2_foxy) ➜  ros2_foxy colcon build to ensure incorrect values are not persisted between runs

There's also an outstanding pyenv issue (CMake might not find the correct python versionbuild --symlink-install --packages-select rcutils --cmake-args=" --log-level=STATUS" --cmake-clean-cache [1.572s] WARNING:colcon.colcon_core.verb:Trigger build Starting >>> rcutils [1.972s] WARNING:colcon.colcon_cmake.task.cmake.build:MAKE [1.973s] WARNING:colcon.colcon_cmake.task.cmake.build:CONFIGURE ['--log-level=STATUS', '-DAMENT_CMAKE_SYMLINK_INSTALL=1'] [1.973s] WARNING:colcon.colcon_cmake.task.cmake.build:================================================================== [1.973s] WARNING:colcon.colcon_cmake.task.cmake.build:context <colcon_core.task.TaskContext object at 0x105ca6f10> [1.974s] WARNING:colcon.colcon_cmake.task.cmake.build:exec ['/usr/local/bin/cmake', '/Users/d/ros2_foxy/src/ros2/rcutils', '--log-level=STATUS', '-DAMENT_CMAKE_SYMLINK_INSTALL=1', '-DCMAKE_INSTALL_PREFIX=/Users/d/ros2_foxy/install/rcutils'] [1.974s] WARNING:colcon.colcon_cmake.task.cmake.build:cwd /Users/d/ros2_foxy/build/rcutils [1.974s] WARNING:colcon.colcon_cmake.task.cmake.build:================================================================== --- stderr: rcutils _python_version_no_dots 37 _python_version 3.7 _library_paths /Users/d/.pyenv/versions/3.7.6/lib/python3.7/config-3.7m-darwin NO_SYSTEM_ENVIRONMENT_PATH Using PythonExtra_LIBRARIES: /Users/d/.pyenv/versions/3.7.6/lib/python3.7/config-3.7m-darwin/libpython3.7m.a --- Finished <<< rcutils [6.88s] Summary: 1 package finished [8.12s] 1 package had stderr output: rcutils

Success! Well, sort of: I still haven't been able to get things running) which might be related.

Answering my own question here:

From FindPythonExtra.cmake I determined the culprit to be some kind of interaction between cmake, python3-config, brew, pyenv, and pipenv.

This was exacerbated by the cmake cache, which persisted incorrectly returned results from python3-config --ldflags between runs.

For future searchers:

First, Note that for debugging, it's useful to pass --cmake-args=" --log-level=STATUS" --cmake-clean-cache to colcon in order to see status messages and avoid caching broken values of the python3-config output between runs.

It appears that when you're using pyenv+pipenv and your virtual environment is pointing to a brew-installed python, python3-config returns incorrect information. For example:

(ros2_foxy) ➜  ros2_foxy which python  
/Users/d/ros2_foxy/.venv/bin/python
/Users/me/ros2_foxy/.venv/bin/python
(ros2_foxy) ➜  ros2_foxy python --version        
Python 3.8.5
(ros2_foxy) ➜  ros2_foxy which python3-config
/Users/d/.pyenv/shims/python3-config
/Users/me/.pyenv/shims/python3-config
(ros2_foxy) ➜  ros2_foxy python3-config --ldflags
-L/Users/d/.pyenv/versions/3.7.6/lib/python3.7/config-3.7m-darwin -L/Users/me/.pyenv/versions/3.7.6/lib/python3.7/config-3.7m-darwin -lpython3.7m -ldl -framework CoreFoundation
(ros2_foxy) ➜  ros2_foxy python3-config --includes

-I/Users/d/.pyenv/versions/3.7.6/include/python3.7m -I/Users/d/.pyenv/versions/3.7.6/include/python3.7m-I/Users/me/.pyenv/versions/3.7.6/include/python3.7m -I/Users/me/.pyenv/versions/3.7.6/include/python3.7m

This results in the ros2 _FindPythonExtra.cmake_ file failing with the error Could NOT find PythonExtra (missing: PythonExtra_LIBRARIES).

On the other hand, if pipenv and pyenv are set up to use a pyenv-installed rather than brew-installed version of Python in the virtual environment, things seem to work. This can be done by explicitly setting an alternate version, e.g. 

➜  ros2_foxy pipenv --python 3.7
Creating a virtualenv for this project…
Pipfile: /Users/d/ros2_foxy/Pipfile
/Users/me/ros2_foxy/Pipfile
Using /Users/d/.pyenv/versions/3.7.6/bin/python3.7m /Users/me/.pyenv/versions/3.7.6/bin/python3.7m (3.7.6) to create virtualenv…
⠏ Creating virtual environment...created virtual environment CPython3.7.6.final.0-64 in 476ms
  creator CPython3Posix(dest=/Users/d/ros2_foxy/.venv, CPython3Posix(dest=/Users/me/ros2_foxy/.venv, clear=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/Users/d/Library/Application app_data_dir=/Users/me/Library/Application Support/virtualenv)
    added seed packages: pip==20.2.1, setuptools==49.6.0, wheel==0.35.1
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator

✔ Successfully created virtual environment! 
Virtualenv location: /Users/d/ros2_foxy/.venv
/Users/me/ros2_foxy/.venv
Creating a Pipfile for this project…

Now the results of python3-config are consistent with the virtual environment:

(ros2_foxy) ➜  ros2_foxy which python
/Users/d/ros2_foxy/.venv/bin/python
/Users/me/ros2_foxy/.venv/bin/python
(ros2_foxy) ➜  ros2_foxy python --version
Python 3.7.6
(ros2_foxy) ➜  ros2_foxy which python3-config
/Users/d/.pyenv/shims/python3-config
/Users/me/.pyenv/shims/python3-config
(ros2_foxy) ➜  ros2_foxy python3-config --ldflags
-L/Users/d/.pyenv/versions/3.7.6/lib/python3.7/config-3.7m-darwin -L/Users/me/.pyenv/versions/3.7.6/lib/python3.7/config-3.7m-darwin -lpython3.7m -ldl -framework CoreFoundation
(ros2_foxy) ➜  ros2_foxy python3-config --includes
-I/Users/d/.pyenv/versions/3.7.6/include/python3.7m -I/Users/d/.pyenv/versions/3.7.6/include/python3.7m
-I/Users/me/.pyenv/versions/3.7.6/include/python3.7m -I/Users/me/.pyenv/versions/3.7.6/include/python3.7m

And rcutils builds:

(ros2_foxy) ➜  ros2_foxy colcon build --symlink-install --packages-select rcutils --cmake-args=" --log-level=STATUS" --cmake-clean-cache
[1.572s] WARNING:colcon.colcon_core.verb:Trigger build
Starting >>> rcutils 
[1.972s] WARNING:colcon.colcon_cmake.task.cmake.build:MAKE
[1.973s] WARNING:colcon.colcon_cmake.task.cmake.build:CONFIGURE ['--log-level=STATUS', '-DAMENT_CMAKE_SYMLINK_INSTALL=1']
[1.973s] WARNING:colcon.colcon_cmake.task.cmake.build:==================================================================
[1.973s] WARNING:colcon.colcon_cmake.task.cmake.build:context         <colcon_core.task.TaskContext object at 0x105ca6f10>
[1.974s] WARNING:colcon.colcon_cmake.task.cmake.build:exec            ['/usr/local/bin/cmake', '/Users/d/ros2_foxy/src/ros2/rcutils', '/Users/me/ros2_foxy/src/ros2/rcutils', '--log-level=STATUS', '-DAMENT_CMAKE_SYMLINK_INSTALL=1', '-DCMAKE_INSTALL_PREFIX=/Users/d/ros2_foxy/install/rcutils']
'-DCMAKE_INSTALL_PREFIX=/Users/me/ros2_foxy/install/rcutils']
[1.974s] WARNING:colcon.colcon_cmake.task.cmake.build:cwd             /Users/d/ros2_foxy/build/rcutils
/Users/me/ros2_foxy/build/rcutils
[1.974s] WARNING:colcon.colcon_cmake.task.cmake.build:==================================================================
--- stderr: rcutils                              
_python_version_no_dots 37
_python_version 3.7
_library_paths /Users/d/.pyenv/versions/3.7.6/lib/python3.7/config-3.7m-darwin
/Users/me/.pyenv/versions/3.7.6/lib/python3.7/config-3.7m-darwin
NO_SYSTEM_ENVIRONMENT_PATH 
Using PythonExtra_LIBRARIES: /Users/d/.pyenv/versions/3.7.6/lib/python3.7/config-3.7m-darwin/libpython3.7m.a
/Users/me/.pyenv/versions/3.7.6/lib/python3.7/config-3.7m-darwin/libpython3.7m.a
---
Finished <<< rcutils [6.88s]

Summary: 1 package finished [8.12s]
  1 package had stderr output: rcutils

Success! Well, sort of: I still haven't been able to get things running

Answering my own question here:

From FindPythonExtra.cmake I determined the culprit to be some kind of interaction between cmake, python3-config, brew, pyenv, and pipenv.

This was exacerbated by the cmake cache, which persisted incorrectly returned results from python3-config --ldflags between runs.

For future searchers:

First, Note that for debugging, it's useful to pass --cmake-args=" --log-level=STATUS" --cmake-clean-cache to colcon in order to see status messages and avoid caching broken values of the python3-config output between runs.

It appears that when you're using pyenv+pipenv and your virtual environment is pointing to a brew-installed python, python3-config returns incorrect information. For example:

(ros2_foxy) ➜  ros2_foxy which python  
/Users/me/ros2_foxy/.venv/bin/python
(ros2_foxy) ➜  ros2_foxy python --version        
Python 3.8.5
(ros2_foxy) ➜  ros2_foxy which python3-config
/Users/me/.pyenv/shims/python3-config
(ros2_foxy) ➜  ros2_foxy python3-config --ldflags
-L/Users/me/.pyenv/versions/3.7.6/lib/python3.7/config-3.7m-darwin -lpython3.7m -ldl -framework CoreFoundation
(ros2_foxy) ➜  ros2_foxy python3-config --includes
-I/Users/me/.pyenv/versions/3.7.6/include/python3.7m -I/Users/me/.pyenv/versions/3.7.6/include/python3.7m

-I/Users/me/.pyenv/versions/3.7.6/include/python3.7m -I/Users/me/.pyenv/versions/3.7.6/include/python3.7m

This results in the ros2 _FindPythonExtra.cmake_ file failing with the error Could NOT find PythonExtra (missing: PythonExtra_LIBRARIES).

On the other hand, if pipenv and pyenv are set up to use a pyenv-installed rather than brew-installed version of Python in the virtual environment, things seem to work. This can be done by explicitly setting an alternate version, e.g. 

➜  ros2_foxy pipenv --python 3.7
Creating a virtualenv for this project…
Pipfile: /Users/me/ros2_foxy/Pipfile
Using /Users/me/.pyenv/versions/3.7.6/bin/python3.7m (3.7.6) to create virtualenv…
⠏ Creating virtual environment...created virtual environment CPython3.7.6.final.0-64 in 476ms
  creator CPython3Posix(dest=/Users/me/ros2_foxy/.venv, clear=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/Users/me/Library/Application Support/virtualenv)
    added seed packages: pip==20.2.1, setuptools==49.6.0, wheel==0.35.1
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator

✔ Successfully created virtual environment! 
Virtualenv location: /Users/me/ros2_foxy/.venv
Creating a Pipfile for this project…

Now the results of python3-config are consistent with the virtual environment:

(ros2_foxy) ➜  ros2_foxy which python
/Users/me/ros2_foxy/.venv/bin/python
(ros2_foxy) ➜  ros2_foxy python --version
Python 3.7.6
(ros2_foxy) ➜  ros2_foxy which python3-config
/Users/me/.pyenv/shims/python3-config
(ros2_foxy) ➜  ros2_foxy python3-config --ldflags
-L/Users/me/.pyenv/versions/3.7.6/lib/python3.7/config-3.7m-darwin -lpython3.7m -ldl -framework CoreFoundation
(ros2_foxy) ➜  ros2_foxy python3-config --includes
-I/Users/me/.pyenv/versions/3.7.6/include/python3.7m -I/Users/me/.pyenv/versions/3.7.6/include/python3.7m

And rcutils builds:

(ros2_foxy) ➜  ros2_foxy colcon build --symlink-install --packages-select rcutils --cmake-args=" --log-level=STATUS" --cmake-clean-cache
[1.572s] WARNING:colcon.colcon_core.verb:Trigger build
Starting >>> rcutils 
[1.972s] WARNING:colcon.colcon_cmake.task.cmake.build:MAKE
[1.973s] WARNING:colcon.colcon_cmake.task.cmake.build:CONFIGURE ['--log-level=STATUS', '-DAMENT_CMAKE_SYMLINK_INSTALL=1']
[1.973s] WARNING:colcon.colcon_cmake.task.cmake.build:==================================================================
[1.973s] WARNING:colcon.colcon_cmake.task.cmake.build:context         <colcon_core.task.TaskContext object at 0x105ca6f10>
[1.974s] WARNING:colcon.colcon_cmake.task.cmake.build:exec            ['/usr/local/bin/cmake', '/Users/me/ros2_foxy/src/ros2/rcutils', '--log-level=STATUS', '-DAMENT_CMAKE_SYMLINK_INSTALL=1', '-DCMAKE_INSTALL_PREFIX=/Users/me/ros2_foxy/install/rcutils']
[1.974s] WARNING:colcon.colcon_cmake.task.cmake.build:cwd             /Users/me/ros2_foxy/build/rcutils
[1.974s] WARNING:colcon.colcon_cmake.task.cmake.build:==================================================================
--- stderr: rcutils                              
_python_version_no_dots 37
_python_version 3.7
_library_paths /Users/me/.pyenv/versions/3.7.6/lib/python3.7/config-3.7m-darwin
NO_SYSTEM_ENVIRONMENT_PATH 
Using PythonExtra_LIBRARIES: /Users/me/.pyenv/versions/3.7.6/lib/python3.7/config-3.7m-darwin/libpython3.7m.a
---
Finished <<< rcutils [6.88s]

Summary: 1 package finished [8.12s]
  1 package had stderr output: rcutils

Success! Well, sort of: I still haven't been able to get things running

Answering my own question here:

From FindPythonExtra.cmake I determined the culprit to be some kind of interaction between cmake, python3-config, brew, pyenv, and pipenv.

This was exacerbated by the cmake cache, which persisted incorrectly returned results from python3-config --ldflags between runs.

For future searchers:

First, Note note that for debugging, it's useful to pass --cmake-args=" --log-level=STATUS" --cmake-clean-cache to colcon in order to see status messages and avoid caching broken values of the python3-config output between runs.

It appears that when you're using pyenv+pipenv and your virtual environment is pointing to a brew-installed python, python3-config returns incorrect information. For example:

(ros2_foxy) ➜  ros2_foxy which python  
/Users/me/ros2_foxy/.venv/bin/python
(ros2_foxy) ➜  ros2_foxy python --version        
Python 3.8.5
(ros2_foxy) ➜  ros2_foxy which python3-config
/Users/me/.pyenv/shims/python3-config
(ros2_foxy) ➜  ros2_foxy python3-config --ldflags
-L/Users/me/.pyenv/versions/3.7.6/lib/python3.7/config-3.7m-darwin -lpython3.7m -ldl -framework CoreFoundation
(ros2_foxy) ➜  ros2_foxy python3-config --includes
-I/Users/me/.pyenv/versions/3.7.6/include/python3.7m -I/Users/me/.pyenv/versions/3.7.6/include/python3.7m

This results in the ros2 _FindPythonExtra.cmake_ file failing with the error Could NOT find PythonExtra (missing: PythonExtra_LIBRARIES).

On the other hand, if pipenv and pyenv are set up to use a pyenv-installed rather than brew-installed version of Python in the virtual environment, things seem to work. This can be done by explicitly setting an alternate version, e.g. 

➜  ros2_foxy pipenv --python 3.7
Creating a virtualenv for this project…
Pipfile: /Users/me/ros2_foxy/Pipfile
Using /Users/me/.pyenv/versions/3.7.6/bin/python3.7m (3.7.6) to create virtualenv…
⠏ Creating virtual environment...created virtual environment CPython3.7.6.final.0-64 in 476ms
  creator CPython3Posix(dest=/Users/me/ros2_foxy/.venv, clear=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/Users/me/Library/Application Support/virtualenv)
    added seed packages: pip==20.2.1, setuptools==49.6.0, wheel==0.35.1
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator

✔ Successfully created virtual environment! 
Virtualenv location: /Users/me/ros2_foxy/.venv
Creating a Pipfile for this project…

Now the results of python3-config are consistent with the virtual environment:

(ros2_foxy) ➜  ros2_foxy which python
/Users/me/ros2_foxy/.venv/bin/python
(ros2_foxy) ➜  ros2_foxy python --version
Python 3.7.6
(ros2_foxy) ➜  ros2_foxy which python3-config
/Users/me/.pyenv/shims/python3-config
(ros2_foxy) ➜  ros2_foxy python3-config --ldflags
-L/Users/me/.pyenv/versions/3.7.6/lib/python3.7/config-3.7m-darwin -lpython3.7m -ldl -framework CoreFoundation
(ros2_foxy) ➜  ros2_foxy python3-config --includes
-I/Users/me/.pyenv/versions/3.7.6/include/python3.7m -I/Users/me/.pyenv/versions/3.7.6/include/python3.7m

And rcutils builds:

(ros2_foxy) ➜  ros2_foxy colcon build --symlink-install --packages-select rcutils --cmake-args=" --log-level=STATUS" --cmake-clean-cache
[1.572s] WARNING:colcon.colcon_core.verb:Trigger build
Starting >>> rcutils 
[1.972s] WARNING:colcon.colcon_cmake.task.cmake.build:MAKE
[1.973s] WARNING:colcon.colcon_cmake.task.cmake.build:CONFIGURE ['--log-level=STATUS', '-DAMENT_CMAKE_SYMLINK_INSTALL=1']
[1.973s] WARNING:colcon.colcon_cmake.task.cmake.build:==================================================================
[1.973s] WARNING:colcon.colcon_cmake.task.cmake.build:context         <colcon_core.task.TaskContext object at 0x105ca6f10>
[1.974s] WARNING:colcon.colcon_cmake.task.cmake.build:exec            ['/usr/local/bin/cmake', '/Users/me/ros2_foxy/src/ros2/rcutils', '--log-level=STATUS', '-DAMENT_CMAKE_SYMLINK_INSTALL=1', '-DCMAKE_INSTALL_PREFIX=/Users/me/ros2_foxy/install/rcutils']
[1.974s] WARNING:colcon.colcon_cmake.task.cmake.build:cwd             /Users/me/ros2_foxy/build/rcutils
[1.974s] WARNING:colcon.colcon_cmake.task.cmake.build:==================================================================
--- stderr: rcutils                              
_python_version_no_dots 37
_python_version 3.7
_library_paths /Users/me/.pyenv/versions/3.7.6/lib/python3.7/config-3.7m-darwin
NO_SYSTEM_ENVIRONMENT_PATH 
Using PythonExtra_LIBRARIES: /Users/me/.pyenv/versions/3.7.6/lib/python3.7/config-3.7m-darwin/libpython3.7m.a
---
Finished <<< rcutils [6.88s]

Summary: 1 package finished [8.12s]
  1 package had stderr output: rcutils

Success! Well, sort of: I still haven't been able to get things running

Answering my own question here:

From FindPythonExtra.cmake I determined the culprit to be some kind of interaction between cmake, python3-config, brew, pyenv, and pipenv.

This was exacerbated by the cmake cache, which persisted incorrectly returned results from python3-config --ldflags between runs.

First, note that for debugging, it's useful to pass --cmake-args=" --log-level=STATUS" --cmake-clean-cache to colcon in order to see status messages and avoid caching broken values of the python3-config output between runs.

It appears that when you're using pyenv+pipenv and your virtual environment is pointing to a brew-installed python, python3-config returns incorrect information. For example:

(ros2_foxy) ➜  ros2_foxy which python  
/Users/me/ros2_foxy/.venv/bin/python
(ros2_foxy) ➜  ros2_foxy python --version        
Python 3.8.5
(ros2_foxy) ➜  ros2_foxy which python3-config
/Users/me/.pyenv/shims/python3-config
(ros2_foxy) ➜  ros2_foxy python3-config --ldflags
-L/Users/me/.pyenv/versions/3.7.6/lib/python3.7/config-3.7m-darwin -lpython3.7m -ldl -framework CoreFoundation
(ros2_foxy) ➜  ros2_foxy python3-config --includes
-I/Users/me/.pyenv/versions/3.7.6/include/python3.7m -I/Users/me/.pyenv/versions/3.7.6/include/python3.7m

This results in the ros2 _FindPythonExtra.cmake_ file failing with the error Could NOT find PythonExtra (missing: PythonExtra_LIBRARIES).

On the other hand, if pipenv and pyenv are set up to use a pyenv-installed rather than brew-installed version of Python in the virtual environment, things seem to work. This can be done by explicitly setting an alternate version, e.g. 

➜  ros2_foxy pipenv --python 3.7
Creating a virtualenv for this project…
Pipfile: /Users/me/ros2_foxy/Pipfile
Using /Users/me/.pyenv/versions/3.7.6/bin/python3.7m (3.7.6) to create virtualenv…
⠏ Creating virtual environment...created virtual environment CPython3.7.6.final.0-64 in 476ms
  creator CPython3Posix(dest=/Users/me/ros2_foxy/.venv, clear=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/Users/me/Library/Application Support/virtualenv)
    added seed packages: pip==20.2.1, setuptools==49.6.0, wheel==0.35.1
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator

✔ Successfully created virtual environment! 
Virtualenv location: /Users/me/ros2_foxy/.venv
Creating a Pipfile for this project…

Now the results of python3-config are consistent with the virtual environment:

(ros2_foxy) ➜  ros2_foxy which python
/Users/me/ros2_foxy/.venv/bin/python
(ros2_foxy) ➜  ros2_foxy python --version
Python 3.7.6
(ros2_foxy) ➜  ros2_foxy which python3-config
/Users/me/.pyenv/shims/python3-config
(ros2_foxy) ➜  ros2_foxy python3-config --ldflags
-L/Users/me/.pyenv/versions/3.7.6/lib/python3.7/config-3.7m-darwin -lpython3.7m -ldl -framework CoreFoundation
(ros2_foxy) ➜  ros2_foxy python3-config --includes
-I/Users/me/.pyenv/versions/3.7.6/include/python3.7m -I/Users/me/.pyenv/versions/3.7.6/include/python3.7m

And rcutils builds:

(ros2_foxy) ➜  ros2_foxy colcon build --symlink-install --packages-select rcutils --cmake-args=" --log-level=STATUS" --cmake-clean-cache
[1.572s] WARNING:colcon.colcon_core.verb:Trigger build
Starting >>> rcutils 
[1.972s] WARNING:colcon.colcon_cmake.task.cmake.build:MAKE
[1.973s] WARNING:colcon.colcon_cmake.task.cmake.build:CONFIGURE ['--log-level=STATUS', '-DAMENT_CMAKE_SYMLINK_INSTALL=1']
[1.973s] WARNING:colcon.colcon_cmake.task.cmake.build:==================================================================
[1.973s] WARNING:colcon.colcon_cmake.task.cmake.build:context         <colcon_core.task.TaskContext object at 0x105ca6f10>
[1.974s] WARNING:colcon.colcon_cmake.task.cmake.build:exec            ['/usr/local/bin/cmake', '/Users/me/ros2_foxy/src/ros2/rcutils', '--log-level=STATUS', '-DAMENT_CMAKE_SYMLINK_INSTALL=1', '-DCMAKE_INSTALL_PREFIX=/Users/me/ros2_foxy/install/rcutils']
[1.974s] WARNING:colcon.colcon_cmake.task.cmake.build:cwd             /Users/me/ros2_foxy/build/rcutils
[1.974s] WARNING:colcon.colcon_cmake.task.cmake.build:==================================================================
--- stderr: rcutils                              
_python_version_no_dots 37
_python_version 3.7
_library_paths /Users/me/.pyenv/versions/3.7.6/lib/python3.7/config-3.7m-darwin
NO_SYSTEM_ENVIRONMENT_PATH 
Using PythonExtra_LIBRARIES: /Users/me/.pyenv/versions/3.7.6/lib/python3.7/config-3.7m-darwin/libpython3.7m.a
---
Finished <<< rcutils [6.88s]

Summary: 1 package finished [8.12s]
  1 package had stderr output: rcutils

Success! Well, sort of: I still haven't been able to get things running

Answering my own question here:

From FindPythonExtra.cmake I determined the culprit to be some kind of interaction between cmake, python3-config, brew, pyenv, and pipenv.

This was exacerbated by the cmake cache, which persisted incorrectly returned results from python3-config --ldflags between runs.

First, note that for debugging, it's useful to pass --cmake-args=" --log-level=STATUS" --cmake-clean-cache to colcon in order to see status messages and avoid caching broken values of the python3-config output between runs.

It appears that when you're using pyenv+pipenv and your virtual environment is pointing to a brew-installed python, python3-config returns incorrect information. For example:

➜ pipenv shell
(ros2_foxy) ➜ which python  
/Users/me/ros2_foxy/.venv/bin/python
(ros2_foxy) ➜ python --version        
Python 3.8.5
(ros2_foxy) ➜ which python3-config
/Users/me/.pyenv/shims/python3-config
(ros2_foxy) ➜ python3-config --ldflags
-L/Users/me/.pyenv/versions/3.7.6/lib/python3.7/config-3.7m-darwin -lpython3.7m -ldl -framework CoreFoundation
(ros2_foxy) ➜ python3-config --includes
-I/Users/me/.pyenv/versions/3.7.6/include/python3.7m -I/Users/me/.pyenv/versions/3.7.6/include/python3.7m

This results in the ros2 _FindPythonExtra.cmake_ file failing with the error Could NOT find PythonExtra (missing: PythonExtra_LIBRARIES).

On the other hand, if pipenv and pyenv are set up to use a pyenv-installed rather than brew-installed version of Python in the virtual environment, things seem to work. This can be done by explicitly setting an alternate version, e.g. 

➜ pipenv shell --python 3.7
Creating a virtualenv for this project…
Pipfile: /Users/me/ros2_foxy/Pipfile
Using /Users/me/.pyenv/versions/3.7.6/bin/python3.7m (3.7.6) to create virtualenv…
⠏ Creating virtual environment...created virtual environment CPython3.7.6.final.0-64 in 476ms
  creator CPython3Posix(dest=/Users/me/ros2_foxy/.venv, clear=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/Users/me/Library/Application Support/virtualenv)
    added seed packages: pip==20.2.1, setuptools==49.6.0, wheel==0.35.1
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator

✔ Successfully created virtual environment! 
Virtualenv location: /Users/me/ros2_foxy/.venv
Creating a Pipfile for this project…

Now the results of python3-config are consistent with the virtual environment:

(ros2_foxy) ➜ which python
/Users/me/ros2_foxy/.venv/bin/python
(ros2_foxy) ➜ python --version
Python 3.7.6
(ros2_foxy) ➜ which python3-config
/Users/me/.pyenv/shims/python3-config
(ros2_foxy) ➜ python3-config --ldflags
-L/Users/me/.pyenv/versions/3.7.6/lib/python3.7/config-3.7m-darwin -lpython3.7m -ldl -framework CoreFoundation
(ros2_foxy) ➜ python3-config --includes
-I/Users/me/.pyenv/versions/3.7.6/include/python3.7m -I/Users/me/.pyenv/versions/3.7.6/include/python3.7m

And rcutils builds:

(ros2_foxy) ➜ colcon build --symlink-install --packages-select rcutils --cmake-args=" --log-level=STATUS" --cmake-clean-cache
[1.572s] WARNING:colcon.colcon_core.verb:Trigger build
Starting >>> rcutils 
[1.972s] WARNING:colcon.colcon_cmake.task.cmake.build:MAKE
[1.973s] WARNING:colcon.colcon_cmake.task.cmake.build:CONFIGURE ['--log-level=STATUS', '-DAMENT_CMAKE_SYMLINK_INSTALL=1']
[1.973s] WARNING:colcon.colcon_cmake.task.cmake.build:==================================================================
[1.973s] WARNING:colcon.colcon_cmake.task.cmake.build:context         <colcon_core.task.TaskContext object at 0x105ca6f10>
[1.974s] WARNING:colcon.colcon_cmake.task.cmake.build:exec            ['/usr/local/bin/cmake', '/Users/me/ros2_foxy/src/ros2/rcutils', '--log-level=STATUS', '-DAMENT_CMAKE_SYMLINK_INSTALL=1', '-DCMAKE_INSTALL_PREFIX=/Users/me/ros2_foxy/install/rcutils']
[1.974s] WARNING:colcon.colcon_cmake.task.cmake.build:cwd             /Users/me/ros2_foxy/build/rcutils
[1.974s] WARNING:colcon.colcon_cmake.task.cmake.build:==================================================================
--- stderr: rcutils                              
_python_version_no_dots 37
_python_version 3.7
_library_paths /Users/me/.pyenv/versions/3.7.6/lib/python3.7/config-3.7m-darwin
NO_SYSTEM_ENVIRONMENT_PATH 
Using PythonExtra_LIBRARIES: /Users/me/.pyenv/versions/3.7.6/lib/python3.7/config-3.7m-darwin/libpython3.7m.a
---
Finished <<< rcutils [6.88s]

Summary: 1 package finished [8.12s]
  1 package had stderr output: rcutils

Success! Well, sort of: I still haven't been able to get things running

Answering my own question here:

From FindPythonExtra.cmake I determined the culprit to be some kind of interaction between cmake, python3-config, brew, pyenv, and pipenv.

This was exacerbated by the cmake cache, which persisted incorrectly returned results from python3-config --ldflags between runs.

First, note that for debugging, it's useful to pass --cmake-args=" --log-level=STATUS" --cmake-clean-cache to colcon in order to see status messages and avoid caching broken values of the python3-config output between runs.

It appears that when you're using pyenv+pipenv and your virtual environment is pointing to a brew-installed python, python3-config returns incorrect information. For example:

➜ pipenv shell
(ros2_foxy) ➜ which python  
/Users/me/ros2_foxy/.venv/bin/python
(ros2_foxy) ➜ python --version        
Python 3.8.5
(ros2_foxy) ➜ which python3-config
/Users/me/.pyenv/shims/python3-config
(ros2_foxy) ➜ python3-config --ldflags
-L/Users/me/.pyenv/versions/3.7.6/lib/python3.7/config-3.7m-darwin -lpython3.7m -ldl -framework CoreFoundation
(ros2_foxy) ➜ python3-config --includes
-I/Users/me/.pyenv/versions/3.7.6/include/python3.7m -I/Users/me/.pyenv/versions/3.7.6/include/python3.7m

This results in the ros2 _FindPythonExtra.cmake_ file failing with the error Could NOT find PythonExtra (missing: PythonExtra_LIBRARIES).

On the other hand, if pipenv and pyenv are set up to use a pyenv-installed rather than brew-installed version of Python in the virtual environment, things seem to work. This can be done by explicitly setting an alternate version, e.g. 

➜ pipenv shell --python 3.7
Creating a virtualenv for this project…
Pipfile: /Users/me/ros2_foxy/Pipfile
Using /Users/me/.pyenv/versions/3.7.6/bin/python3.7m (3.7.6) to create virtualenv…
⠏ Creating virtual environment...created virtual environment CPython3.7.6.final.0-64 in 476ms
  creator CPython3Posix(dest=/Users/me/ros2_foxy/.venv, clear=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/Users/me/Library/Application Support/virtualenv)
    added seed packages: pip==20.2.1, setuptools==49.6.0, wheel==0.35.1
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator

✔ Successfully created virtual environment! 
Virtualenv location: /Users/me/ros2_foxy/.venv
Creating a Pipfile for this project…

Now the results of python3-config are consistent with the virtual environment:

(ros2_foxy) ➜ which python
/Users/me/ros2_foxy/.venv/bin/python
(ros2_foxy) ➜ python --version
Python 3.7.6
(ros2_foxy) ➜ which python3-config
/Users/me/.pyenv/shims/python3-config
(ros2_foxy) ➜ python3-config --ldflags
-L/Users/me/.pyenv/versions/3.7.6/lib/python3.7/config-3.7m-darwin -lpython3.7m -ldl -framework CoreFoundation
(ros2_foxy) ➜ python3-config --includes
-I/Users/me/.pyenv/versions/3.7.6/include/python3.7m -I/Users/me/.pyenv/versions/3.7.6/include/python3.7m

And rcutils builds:

(ros2_foxy) ➜ colcon build --symlink-install --packages-select rcutils --cmake-args=" --log-level=STATUS" --cmake-clean-cache
[1.572s] WARNING:colcon.colcon_core.verb:Trigger build
Starting >>> rcutils 
[1.972s] WARNING:colcon.colcon_cmake.task.cmake.build:MAKE
[1.973s] WARNING:colcon.colcon_cmake.task.cmake.build:CONFIGURE ['--log-level=STATUS', '-DAMENT_CMAKE_SYMLINK_INSTALL=1']
[1.973s] WARNING:colcon.colcon_cmake.task.cmake.build:==================================================================
[1.973s] WARNING:colcon.colcon_cmake.task.cmake.build:context         <colcon_core.task.TaskContext object at 0x105ca6f10>
[1.974s] WARNING:colcon.colcon_cmake.task.cmake.build:exec            ['/usr/local/bin/cmake', '/Users/me/ros2_foxy/src/ros2/rcutils', '--log-level=STATUS', '-DAMENT_CMAKE_SYMLINK_INSTALL=1', '-DCMAKE_INSTALL_PREFIX=/Users/me/ros2_foxy/install/rcutils']
[1.974s] WARNING:colcon.colcon_cmake.task.cmake.build:cwd             /Users/me/ros2_foxy/build/rcutils
[1.974s] WARNING:colcon.colcon_cmake.task.cmake.build:==================================================================
--- stderr: rcutils                              
_python_version_no_dots 37
_python_version 3.7
_library_paths /Users/me/.pyenv/versions/3.7.6/lib/python3.7/config-3.7m-darwin
NO_SYSTEM_ENVIRONMENT_PATH 
Using PythonExtra_LIBRARIES: /Users/me/.pyenv/versions/3.7.6/lib/python3.7/config-3.7m-darwin/libpython3.7m.a
---
Finished <<< rcutils [6.88s]

Summary: 1 package finished [8.12s]
  1 package had stderr output: rcutils

Success! Well, sort of: I still haven't been able to get things running

ROS Answers is licensed under Creative Commons Attribution 3.0 Content on this site is licensed under a Creative Commons Attribution Share Alike 3.0 license.
about | faq | help | privacy policy | terms of service
Powered by Askbot version 0.10.2