Ask Your Question
0

How to setup a buildfarm for my git repo ?

asked 2018-04-23 14:47:28 -0600

venkisagunner gravatar image

updated 2018-04-27 15:05:30 -0600

NOTE: I'm asking this question after going through series of web pages, ros discourse discussions, ros answers discussions and issues in github. Please note that I made a considerable effort to learn but the documentation to me seems more sparse (high level). Please understand that I'm a total noob and if something you see here looks rubbish or garbage please point it out. I'm ready to learn. Having said that here it goes.

  • What exactly I'm trying to achieve here ?

I have a git repo with me that takes in the camera images and estimates user pose, saves data and trains a classifier. I want to setup a build farm for this repo, perform continuous integration and post build steps. Oh ! and also, please let me know if this is possible with the ros buildfarm. Provide documentation that clearly explains what to do if someone like me wants to setup a buildfarm.

  • What have I done so far ?

I started off here at ros.org to learn about buildfarms and then I stumbled upon buildfarm_deployment which I tried to follow in the following ways.

  1. Fork the repo, create ssh keys in my master machine which is ubuntu 16.04 xenial and has ros kinetic which is running on a VM. I thought of having a Jetson Tx2 as my agent and a Raspberry Pi 3 B as my repo but I'm willing to change this to two more VMs having same specs as my master. Please let me know whether there are any other steps apart from setting up 3 instances which is not captured in the documentation that I have to follow.
  2. I found that the next step is to change the common.yaml file which is found here which I directly edited on my forked repo. I have several questions on this one. Any good piece of information will be greatly appreciated.

    2.1. In common.yaml file, with the comments for line5-6, I understood that I have to replace the values for jenkins::slave::ui_user and jenkins::slave::ui_pass with my jenkins username & password and I changed it. If this wrong please advise me what exactly I should be doing here.

    2.2. Next up I left line-9 as it is. Lines 11-12 were simple enough that I changed the IP address to my master machine and raspi (for now, I can change this to another VM if cross compilation won't work). Line-15 is time zone and I changed it accordingly.

    2.3. Lines 17-23, I would like to know how to actually do this. First up, I generated a ssh keygen based on this link, which should generate two keys, a public and a private one. In line-21, should I copy paste the public key or the private key, I generated locally ?

Once I get the answers for all the questions, I will proceed with the next set of questions. Thank you for taking time in reading this.

EDIT - 1 ... (more)

edit retag flag offensive close merge delete

2 Answers

Sort by » oldest newest most voted
1

answered 2018-04-23 18:36:46 -0600

nuclearsandwich gravatar image

updated 2018-04-27 16:22:18 -0600

What exactly I'm trying to achieve here ?

I have a git repo with me that takes in the camera images and estimates user pose, saves data and trains a classifier. I want to setup a build farm for this repo, perform continuous integration and post build steps. Oh ! and also, please let me know if this is possible with the ros buildfarm. Provide documentation that clearly explains what to do if someone like me wants to setup a buildfarm.

Depending on your end-goal, it may be easier to begin either without ros_buildfarm or using ros_buildfarm without a full buildfarm deployment:

A. Running your repositories tests.
If all you want is basic continuous testing, setting up an entire buildfarm might be overkill. There are examples of setting up ROS repositories in Travis. This repository provides a sample .travis.yml and information for doing exactly that. It might be a bit stale and it's not something I've worked with, but once set up, it would run your tests without the need for running your own buildfarm. MoveIt! has a CI repository with an up-to-date Travis config as well. (Hat tip to @Geoff in the comments).

B. If you also want to build binary packages for your repository, it's still possible to do without setting up a buildfarm deployment, but you will need to create suitable rosdistro and ros_buildfarm_config configuration files so that the ros_buildfarm scripts can be run on your local machines.

C. Of course, you can also set up a complete buildfarm cluster to perform run ros_buildfarm operations. This is probably the best option if you're going to be adding more custom packages down the road or building for multiple rosdistros. It's also the course of action you're already on.


ARM build agents

I thought of having a Jetson Tx2 as my agent and a Raspberry Pi 3 B as my repo but I'm willing to change this to two more VMs having same specs as my master.

I mentioned in your previous question that the buildfarm_deployment repository was originally written for x86 hosts only and that I know it doesn't support arm64 hosts out of the box and it probably won't work on a Raspberry Pi running Raspbian Stretch or even Ubuntu Xenial. I have deployed arm64 agents but I had to do the last 40% of the setup by hand which requires some experience and understanding of what the Puppet config management is doing for you so it won't be straightforward if you're new to Puppet and ros_buildfarm. You also won't be able to build for AMD64 (x86_64) targets with an arm64 agent, though you can build for arm64 with an AMD64 host using qemu, which the buildfarm_deployment sets up by default.

I would recommend using AMD64 virtual machines although they don't all need to be as powerful as your Jenkins master. This document has some recommended instances sizes using AWS instances sizes as a ... (more)

edit flag offensive delete link more

Comments

For Travis integration the MoveIt! CI repository has been updated recently so is likely to still work.

Geoff gravatar imageGeoff ( 2018-04-23 20:53:36 -0600 )edit

Thank you for your detailed explanation. I'm following the steps you mentioned and yes, I'm building a buildfarm (C). I would like to know whether I have to install jenkins in my VM instance ?

venkisagunner gravatar imagevenkisagunner ( 2018-04-25 16:58:07 -0600 )edit

I would like to know whether I have to install jenkins in my VM instance ?

The buildfarm_deployment uses Puppet to install jenkins for you, it should work as long as your VM is relatively clean. Deployment instructions: https://github.com/ros-infrastructure...

nuclearsandwich gravatar imagenuclearsandwich ( 2018-04-25 17:00:05 -0600 )edit

I took the instructions and I have configured the common.yaml file accordingly. I have added a new edit to my question. Hope you could help @nuclearsandwich. Thanks !

venkisagunner gravatar imagevenkisagunner ( 2018-04-27 15:07:37 -0600 )edit

Puppet (err): Evaluation Error: Error while evaluating a Function Call, Error from DataBinding 'hiera' while looking up 'ntp::autoupdate': (<unknown>): could not find expected ':' while scanning a simple key at line 33 column 1 at /root/buildfarm_deployment/modules/profile/manifests/ros/base.pp:8:3

venkisagunner gravatar imagevenkisagunner ( 2018-05-03 12:12:24 -0600 )edit

I'm getting an error like the above one when I tried to run reconfigure bash script for master. Could you please help me out here ?

venkisagunner gravatar imagevenkisagunner ( 2018-05-03 12:13:41 -0600 )edit

The question is quite old - but I stumbled over the same issue, so it might help others. That is most likely an error in some yaml file. Look here https://answers.ros.org/question/2818... Download yamllint. check common.yaml, ...

klausd gravatar imageklausd ( 2018-11-16 09:52:01 -0600 )edit
0

answered 2018-04-27 14:42:00 -0600

Dirk Thomas gravatar image

If not running your own infrastructure is an option (similar to option A suggested by @nuclearsandwich) you could setup Travis CI (or any other CI provider) to build and test your code. The ros_buildfarm docs contain some example and snippets how to do so: e.g. https://github.com/ros-infrastructure...

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

3 followers

Stats

Asked: 2018-04-23 14:47:28 -0600

Seen: 629 times

Last updated: Apr 27 '18