How to disable certain tests in ROS 2 Buildfarm?
We are running system tests and a CI has to include dependencies that are not available in rosdep. Since we are already running the system tests on our CI machines I was wondering whether we can just disable them in ROS 2 Buildfarm? Getting emails such as Jenkins build is still unstable is a bit annoying. Maybe there is an environment variable specific to ROS 2 Buildfarm that we can use to ignore some tests?
Asked by lukicdarkoo on 2021-09-23 07:19:53 UTC
Answers
Would removing the dev job for your repository be an option?
That may work, but then I will not get notified if a build fails?
Yes, the testing jobs. (PR and dev) are completely separate processes from the actual builds (binarydeb etc) and you will still get email about them failing if they fail. Disabling the testing jobs will avoid the notifications of being unstable due to tests failing by not running them at all.
Our system tests are in a separate package (but in the common repository), would you recommend not releasing it?
In general if the system tests are not designed to be used by downstream users, we recommend not releasing them. The test packages don't have a lot of value to be gained from being installable individually and running against the binary releases installed on the system as the binary installation should have been QAd before release.
You can add a ignored list in the release repsitory such as: https://github.com/ros-gbp/ros_comm-release/blob/master/noetic.ignored
Asked by tfoote on 2021-09-23 12:27:05 UTC
Comments
Would removing the dev job for your repository be an option? That's the
sourcestanza in thedistribution.yamlof the ROS distribution you target.The answer to the question is "you can't", I believe.
Asked by gvdhoorn on 2021-09-23 07:33:17 UTC
That may work, but then I will not get notified if a build fails? Our system tests are in a separate package (but in the common repository), would you recommend not releasing it?
Asked by lukicdarkoo on 2021-09-23 08:12:20 UTC