ROS BuildFarm with Private GitHub Repositories

edit

How do I get the “generate_release_script.py” to generate a script that can properly pull code from my private GitHub repo?

generate_release_script.py creates a script that runs local containers for generating source packages and binary packages from a release repository, which on a Jenkins build farm with jobs created by generate_release_job.py would be broken up into sourcepkg/sourcedeb jobs and binarydeb/binarypkg jobs. Only the sourcedeb job on jenkins or sourcedeb job on the script actually needs access to the package release repository since after the source package is created the generated source package data is hosted on the build farm repository. In the release script the source package is passed via shared docker mount point between the source and release phases.

When git_ssh_credential_id is specified, the running SSH_AUTH_SOCK environment variable is set for the container runtime [1]. When sourcepkg jobs are run on Jenkins, setting the git_ssh_credential_id parameter will make the ssh key in that credential available to the job process via ssh agent, which the docker-run invocation is sharing into the container by mounting the SSH_AUTH_SOCK and setting the environment variable [2].

To use a repository requiring authentication locally when no actual Jenkins build farm is configured, your ros_buildfarm_config should still set git_ssh_credential_id even if the value is something arbitrary like ssh_agent. When running the generated release script make sure that an ssh key which can pull from the target release repository has been added to your local ssh-agent since that is the agent which will be mounted and exposed to the running sourcepkg container.

Also, to this point I have been leaving my rosdistro repository fully public, but that’s not ideal, and would like to make that private as well. How do I do that without having to fully self-host all the repositories myself?

Most of the jobs on the buildfarm will use the same git_ssh_credential_id to for remote git operations in much the same way as the sourcepkg job does as described above.

All I really want to do is build debian packages for my ROS packages. When I attempted to use the bloom debian generator I noticed that it did not handle dependencies, and so I found myself manually building each debian in order.

Bloom's primary responsibility is using rosdep, rosdistro, and package.xml sources to render package metadata into the packaging format required by different platforms ROS supports, primarily deb packages for Ubuntu and Debian, and rpm packages for RHEL and Fedora. When you are bringing up a new rosdistro, packages must be bloomed in topological order since each package will try and render its dependencies from the rosdep database and the rosdistro distribution.yaml. Once all packages have been bloomed the first time, future package releases can be bloomed in any order since the packages they depend on will all already be in your rosdistro. You can also avoid having to bloom in topological order the first time by manually creating a "completed" rosdistro with each released package you ... (more)