Ask Your Question

How to add a new distro to a ROS build farm?

asked 2018-12-05 12:45:02 -0600

I've got an existing local build farm that has been building Indigo / Kinetic packages for a while now, and I want to add Melodic to it. Is the upgrade process documented anywhere?

I've tried to figure it out myself but am running into some problems. So far I have:

  1. Edited the repo.yaml file in my buildfarm_deployment_config repository to add "bionic" to the list of distros in my building/testing/main repositories and the ros_bootstraml.yaml file
  2. Added doc/release/source yaml files into a "melodic" directory in my ros_buildfarm_config repository, then referenced them in the index.yaml file
  3. Run buildfarm_config/reconfigure.bash on my repository machine
  4. Run the ros_buildfarm script on my new ros_buildfarm_config

That was enough to create all of the M* jobs in the Manage tab in Jenkins, as well as the M* tabs.

I would expect the next step I need to run is to use the import_upstream job with the ros_bootstramp.yaml config file, but it fails like so:

# BEGIN SUBSECTION: import debian packages for ubuntu_building
11:19:04 + python -u /home/jenkins-agent/workspace/import_upstream/reprepro-updater/scripts/ ubuntu_building /home/jenkins-agent/reprepro_config/ros_bootstrap.yaml --commit
11:19:04 loading config file: /home/jenkins-agent/reprepro_config/ros_bootstrap.yaml
11:19:04 adding arches set(['amd64', 'source', 'i386', 'armel', 'armhf', 'arm64']) and suites: set(['trusty', 'bionic', 'xenial'])
11:19:04 target_distros ['bionic', 'trusty', 'xenial']
11:19:04 target_arches ['bionic', 'trusty', 'xenial']
11:19:04 Updating for bionic amd64 to update into repo /var/repos/ubuntu/building
11:19:04 I have a lock on /var/repos/ubuntu/building/lock
11:19:04 Creating updates file /var/repos/ubuntu/building/conf/updates
11:19:04   Name: ros_bootstrap
11:19:04   Method:
11:19:04   Suite: bionic
11:19:04   Components: main
11:19:04   Architectures: amd64
11:19:04   VerifyRelease: blindtrust
11:19:04 Creating distributions file /var/repos/ubuntu/building/conf/distributions
11:19:04 running command ['reprepro', '-v', '-b', '/var/repos/ubuntu/building', '--noskipold', 'update', 'bionic']
11:19:05 No distribution definition of 'bionic' found in '/var/repos/ubuntu/building/conf/distributions'!
11:19:05 There have been errors!
11:19:06 Execution of [['reprepro', '-v', '-b', '/var/repos/ubuntu/building', '--noskipold', 'update', 'bionic']] Failed: Command '['reprepro', '-v', '-b', '/var/repos/ubuntu/building', '--noskipold', 'update', 'bionic']' returned non-zero exit status 249
11:19:06 Traceback (most recent call last):
11:19:06   File "/home/jenkins-agent/workspace/import_upstream/reprepro-updater/scripts/", line 68, in <module>
11:19:06     updates_generator, distro, arch, options.commit)
11:19:06   File "/home/jenkins-agent/workspace/import_upstream/reprepro-updater/src/reprepro_updater/", line 185, in run_update
11:19:06     raise RuntimeError('update command failed')
11:19:06 RuntimeError: update command failed
11:19:06 Build step 'Execute shell' marked build as failure
11:19:06 Finished: FAILURE

After running it, I can verify that the file at /var/repos/ubuntu/building/conf/distributions only has entries for trusty and ... (more)

edit retag flag offensive close merge delete


I figured out that the configuration for that script is in ~jenkins-agent/.buildfarm/reprepro_updater.ini . I manually updated it and it then successfully ran and created the repository. I would have expected that file to be updated by running reconfigure.bash, but maybe I'm still missing something.

preed gravatar imagepreed ( 2018-12-05 15:16:22 -0600 )edit

Re-running reconfigure.bash is actually not recommended, as is also mentioned in the docs of buildfarm_deployment_config and in other places.

I don't necessarily know how to do what you wanted to do though. I assume @nuclearsandwich will respond with some suggestions at a later time.

gvdhoorn gravatar imagegvdhoorn ( 2018-12-05 15:24:39 -0600 )edit

I don't see anything in the documentation for buildfarm_deployment_config or buildfarm_deployment that advises against re-running reconfigure.bash; in fact, that seems at odds with remarks inside the script itself that refer to re-running it. What's the right way to reconfigure a machine?

preed gravatar imagepreed ( 2018-12-05 15:35:20 -0600 )edit

Can't find the reference in the docs right now, but see ros-infrastructure/buildfarm_deployment#160 fi.

gvdhoorn gravatar imagegvdhoorn ( 2018-12-05 15:39:03 -0600 )edit

The repository and agent roles are safe to reconfigure. #160 only applies to master. and have autoreconfigure set for both. Although usually when I make a change to repo hosts I'm too impatient to wait for it to auto run and do a git pull and ./reconfigure myself.

nuclearsandwich gravatar imagenuclearsandwich ( 2018-12-12 08:11:40 -0600 )edit

1 Answer

Sort by ยป oldest newest most voted

answered 2018-12-12 08:36:40 -0600

nuclearsandwich gravatar image

This is a good question and reinforces my desire to add operations manuals to the buildfarm_deployment repositories. When adding new debian or ubuntu distributions, the first step is, as you surmised, to add them to the reprepro_updater_config and ros_bootstrap.yaml in the repo hiera config.

I just went through the PRs to our private configs to review when we added stretch, bionic, and buster and nothing out-of-the-ordinary was there that I'm forgetting.

My first thought is that there could be puppet errors when reconfiguring. mving or rotating the /var/log/puppet.log file so you don't have to worry about finding the break between puppet runs and triggering a reconfigure might surface any errors that aren't fatal. If there's sensitive information in your puppet logs you don't wish to share on ROS Answers we can sort something out privately and report our findings back with sanitized examples.

I looked through ours and found an old log section that looks like it would make trouble. The python-configparser package should be pulled in by current buildfarm_deployment sources.

2017-06-14 17:03:45 +0000 /Stage[main]/Profile::Ros::Repo/File[/home/jenkins-agent/reprepro_config]/ensure (notice): created 2017-06-14 17:03:45 +0000 /Stage[main]/Profile::Ros::Repo/Exec[init_building_repo]/returns (notice): Traceback (most recent call last): 2017-06-14 17:03:45 +0000 /Stage[main]/Profile::Ros::Repo/Exec[init_building_repo]/returns (notice): File "/home/jenkins-agent/reprepro-updater/scripts/", line 8, in <module> 2017-06-14 17:03:45 +0000 /Stage[main]/Profile::Ros::Repo/Exec[init_building_repo]/returns (notice): from reprepro_updater import conf 2017-06-14 17:03:45 +0000 /Stage[main]/Profile::Ros::Repo/Exec[init_building_repo]/returns (notice): File "/home/jenkins-agent/reprepro-updater/src/reprepro_updater/", line 33, in <module> 2017-06-14 17:03:45 +0000 /Stage[main]/Profile::Ros::Repo/Exec[init_building_repo]/returns (notice): from configparser import SafeConfigParser 2017-06-14 17:03:45 +0000 /Stage[main]/Profile::Ros::Repo/Exec[init_building_repo]/returns (notice): ImportError: No module named configparser 2017-06-14 17:03:45 +0000 Puppet (err): /home/jenkins-agent/reprepro-updater/scripts/ ubuntu_building -c returned 1 instead of one of [0]

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools



Asked: 2018-12-05 12:45:02 -0600

Seen: 107 times

Last updated: Dec 12 '18