Hello Ladies and Gentlemen

I apologize for this long post but, I reckon people will request the info anyway...

I've been struggling for some time, to make ROS behave like I want.
Either my mindset is not suited for ROS or the opposite is the case.
Maybe ROS is not mature enough for real products - I need some help

The setup:

We make agro-rotbots. We have a product with 3-31 target-computers on a weeding/hoeing machine, each running a monolithic piece of legacy software. To facilitate further feature development we need to modularize (“objectify”) the software, and to get there; we need a distributed framework.
We started out with the intention of rolling our own. But then some clever guy pointed our attention to ROS – and it has every thing we need, events, common message interfaces, debug opportunities...... - all on a BSD license could it be any better ??

Then I set out to get a simple proof-of-concept-system running on our target – to see if we should go the ROS way.

To my big surprise it has proven very difficult to build ROS from source the way I want.

I tried but couldn't make it work - what am I doing wrong ? I'm sure there is a very basic thing I missed.

The goal:
1. create a root_fs image that can be copied/'burned' directly on to the target, flash.
(e.i not a 'linux-distribution' - There is no such thing as 'building' or 'installing' on the target)
2. Create a Linux flavored develop environment in that is manageable, and relocatable.

My setup:
My Build computer:
(this is just mine but it could/should be any Linux 32/64 bit combi)
intel x64 (64 bit) Linux version 3.19.0-47-generic (buildd@lgw01-19) (gcc version 4.8.2 (Ubuntu 4.8.2-19ubuntu1) ) £53~14.04.1-Ubuntu SMP Mon Jan 18 16:09:14 UTC 2016
PLEASE NOTE: at the moment ROS is *NOT** installed on my build computer (to avoid 'contamination')*

The target system:
intel x86 (32 bit) some minimal Linux kernel (what ever version we need). the absolute minimal tool set needed.

All our code is under git control

So we need to build a toolchain, cross compile the kernel, ROS, and all our proprietary software.

I have been trying the yocto project with no success, it is much to big and unhandy, for every day use. And we don't want a 'distro' just a plain image

So I went with buildroot since it is simple (in the sense that only uses standard makefiles) and is highly flexible.

I did this:

First I build a clean rootfs that can boot on the target: Linux kernel, with the things ROS needs, boost, python, libtinyxml....
Then I want to compile roscomm using the newly created toolchain, and overlay the root_fs.(Latter I will make a proper buildroot “package”)

Then I fetch and build the ROS software.

Following : http://answers.ros.org/question/191070/compile-roscore-for-arm-board/#194304 and http://wiki.ros.org/xxx/Installation/Source

cd ~
mkdir ros_catkin_ws 
cd ros_catkin_ws 

sudo sh -c 'echo "deb http://packages.ros.org/ros/ubuntu $(lsb_release -sc) main" > /etc/apt/sources.list.d/ros-latest.list'
sudo apt-key adv --keyserver hkp://pool.sks-keyservers.net --recv-key 0xB01FA116
sudo apt-get update
sudo apt-get install python-rosdep python-rosinstall-generator python-wstool python-rosinstall build-essential

# set the type of ROS we want 
#VARIANT=desktop_full  
#VARIANT=desktop
VARIANT=ros_comm

#RELEASE=indigo
#RELEASE=hydro
RELEASE=jade

rosinstall_generator ${VARIANT} --rosdistro ${RELEASE} --deps --wet-only --tar > ${RELEASE}-${VARIANT}-wet.rosinstall  
wstool init -j8 src ${RELEASE}-${VARIANT}-wet.rosinstall  
rosdep install --from-paths src --ignore-src --rosdistro ${RELEASE} -y

So far so good now the fun begins:

./src/catkin/bin/catkin_make_isolated --install -DCMAKE_BUILD_TYPE=Release 

this works fine and make a nice ROS thing in ../install_isolated , for a native x64 build.
1.st. question: why isn't it relocatable all setup files contain absolute path's, that mach my computer only ?

But when I supply my toolchain build by buildroot:

rostoolchain.cmake:

set(CMAKE_SYSTEM_NAME Linux)
set(CMAKE_C_COMPILER  /home/lars/toolchain/host/usr/bin/i686-buildroot-linux-uclibc-gcc )
set(CMAKE_CXX_COMPILER /home/lars/toolchain/host/usr/bin/i686-buildroot-linux-uclibc-g++)
set(CMAKE_FIND_ROOT_PATH /home/lars/toolchain/staging)

# Have to set this one to BOTH, to allow CMake to find rospack
set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM BOTH)
set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY)

I get the following

# ./src/catkin/bin/catkin_make_isolated -DCMAKE_TOOLCHAIN_FILE=${HOME}/repository/ros-buildroot/rostoolchain.cmake
Base path: /home/lars/ros_catkin_ws
Source space: /home/lars/ros_catkin_ws/src
Build space: /home/lars/ros_catkin_ws/build_isolated
Devel space: /home/lars/ros_catkin_ws/devel_isolated
Install space: /home/lars/ros_catkin_ws/install_isolated
Additional CMake Arguments: -DCMAKE_TOOLCHAIN_FILE=/home/lars/repository/ros-buildroot/rostoolchain.cmake
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~~  traversing 50 packages in topological order:
~~  - catkin

yes yes all 50 of them....

~~  - rostopic
~~  - roswtf
~~  - topic_tools
~~  - rosbag
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The packages or cmake arguments have changed, forcing cmake invocation

==> Processing catkin package: 'catkin'
==> Creating build directory: 'build_isolated/catkin'
==> cmake /home/lars/ros_catkin_ws/src/catkin -DCATKIN_DEVEL_PREFIX=/home/lars/ros_catkin_ws/devel_isolated/catkin -DCMAKE_INSTALL_PREFIX=/home/lars/ros_catkin_ws/install_isolated -DCMAKE_TOOLCHAIN_FILE=/home/lars/repository/ros-buildroot/rostoolchain.cmake -G Unix Makefiles in '/home/lars/ros_catkin_ws/build_isolated/catkin'
-- The C compiler identification is GNU 4.9.3
-- The CXX compiler identification is GNU 4.9.3
-- Check for working C compiler: /home/lars/toolchain/host/usr/bin/i686-buildroot-linux-uclibc-gcc
-- Check for working C compiler: /home/lars/toolchain/host/usr/bin/i686-buildroot-linux-uclibc-gcc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: /home/lars/toolchain/host/usr/bin/i686-buildroot-linux-uclibc-g++
-- Check for working CXX compiler: /home/lars/toolchain/host/usr/bin/i686-buildroot-linux-uclibc-g++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Using CATKIN_DEVEL_PREFIX: /home/lars/ros_catkin_ws/devel_isolated/catkin
-- Using CMAKE_PREFIX_PATH: 
-- Found PythonInterp: /usr/bin/python (found version "2.7.6") 
-- Using PYTHON_EXECUTABLE: /usr/bin/python
-- Using Debian Python package layout
-- Using empy: /usr/bin/empy
-- Using CATKIN_ENABLE_TESTING: ON
-- Call enable_testing()
-- Using CATKIN_TEST_RESULTS_DIR: /home/lars/ros_catkin_ws/build_isolated/catkin/test_results
-- Looking for include file pthread.h
-- Looking for include file pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE  
-- Found gtest sources under '/usr/src/gtest': gtests will be built
-- Using Python nosetests: /usr/bin/nosetests-2.7
-- catkin 0.6.16
-- Configuring done
-- Generating done
-- Build files have been written to: /home/lars/ros_catkin_ws/build_isolated/catkin
==> make -j8 -l8 in '/home/lars/ros_catkin_ws/build_isolated/catkin'
<== Finished processing package [1 of 50]: 'catkin'

==> Processing catkin package: 'genmsg'
==> Creating build directory: 'build_isolated/genmsg'
==> Building with env: '/home/lars/ros_catkin_ws/devel_isolated/catkin/env.sh'
==> cmake /home/lars/ros_catkin_ws/src/genmsg -DCATKIN_DEVEL_PREFIX=/home/lars/ros_catkin_ws/devel_isolated/genmsg -DCMAKE_INSTALL_PREFIX=/home/lars/ros_catkin_ws/install_isolated -DCMAKE_TOOLCHAIN_FILE=/home/lars/repository/ros-buildroot/rostoolchain.cmake -G Unix Makefiles in '/home/lars/ros_catkin_ws/build_isolated/genmsg'
-- The C compiler identification is GNU 4.9.3
-- The CXX compiler identification is GNU 4.9.3
-- Check for working C compiler: /home/lars/toolchain/host/usr/bin/i686-buildroot-linux-uclibc-gcc
-- Check for working C compiler: /home/lars/toolchain/host/usr/bin/i686-buildroot-linux-uclibc-gcc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: /home/lars/toolchain/host/usr/bin/i686-buildroot-linux-uclibc-g++
-- Check for working CXX compiler: /home/lars/toolchain/host/usr/bin/i686-buildroot-linux-uclibc-g++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
CMake Error at CMakeLists.txt:3 (find_package):
  By not providing "Findcatkin.cmake" in CMAKE_MODULE_PATH this project has
  asked CMake to find a package configuration file provided by "catkin", but
  CMake did not find one.

  Could not find a package configuration file provided by "catkin" with any
  of the following names:

    catkinConfig.cmake
    catkin-config.cmake

  Add the installation prefix of "catkin" to CMAKE_PREFIX_PATH or set
  "catkin_DIR" to a directory containing one of the above files.  If "catkin"
  provides a separate development package or SDK, be sure it has been
  installed.


-- Configuring incomplete, errors occurred!
See also "/home/lars/ros_catkin_ws/build_isolated/genmsg/CMakeFiles/CMakeOutput.log".
<== Failed to process package 'genmsg': 
  Command '['/home/lars/ros_catkin_ws/devel_isolated/catkin/env.sh', 'cmake', '/home/lars/ros_catkin_ws/src/genmsg', '-DCATKIN_DEVEL_PREFIX=/home/lars/ros_catkin_ws/devel_isolated/genmsg', '-DCMAKE_INSTALL_PREFIX=/home/lars/ros_catkin_ws/install_isolated', '-DCMAKE_TOOLCHAIN_FILE=/home/lars/repository/ros-buildroot/rostoolchain.cmake', '-G', 'Unix Makefiles']' returned non-zero exit status 1

Reproduce this error by running:
==> cd /home/lars/ros_catkin_ws/build_isolated/genmsg && /home/lars/ros_catkin_ws/devel_isolated/catkin/env.sh cmake /home/lars/ros_catkin_ws/src/genmsg -DCATKIN_DEVEL_PREFIX=/home/lars/ros_catkin_ws/devel_isolated/genmsg -DCMAKE_INSTALL_PREFIX=/home/lars/ros_catkin_ws/install_isolated -DCMAKE_TOOLCHAIN_FILE=/home/lars/repository/ros-buildroot/rostoolchain.cmake -G 'Unix Makefiles'

Command failed, exiting.

WFT !! WHAT !!! WHY!!!

When I add the following line to my toolchain.cmake (a clear hack, something else must be seriously wrong ?)

set(catkinDIR /home/lars/roscatkinws/develisolated/catkin/share/catkin/cmake)

it just brings me one step further

Failed to process package 'geneus':

I continue to add set( pgk_DIR,,,) statements to my toolchain.cmake file - untill

Could not find a package configuration file provided by "console_bridge"
  with any of the following names:

    console_bridgeConfig.cmake
    console_bridge-config.cmake

consolebridge is nowhere in my system a little investigation brings me to the conclusion that consolebridge is simply missing from the roscomm source generated by rosinstallgenerator

Tried with desktop and desktop-full variants of jade and get the very same thing

What to do ??? Googling a around provides no solutions..

Is no one using ROS in an real-world cross compiled environment ?

I hope some body out there has buildroot or similar experience, and can give me some pointers, if not I have to revert to the homegrown solution. Best regards Lars Larsen