Using unreleased test dependencies on the ROS build farm

edit

@tfoote: REP-141 seems to imply that doc jobs do support expressing dependency on unreleased packages.

Could that support not be reused for devel jobs?

The other workaround is that the test build actually fetches the resources

Aah, I never though of that, it seems CMake can download a git repo and add it to the project.

On the topic of should this be possible @v4nh notes (on discord) that:

From my perspective test dependencies should also work from released packages, otherwise this is still a bug. I (as a user) would expect to have tests succeed if I download a release and all released dependencies

How would a user run tests with a released package?

The build script -- which contains/sets up the test driver -- is not included in a binary package.

How would a user run tests with a released package?

You are always running tests on a package from source. But they can rely on the installed artifacts of any other package. For example if you need to play a rosbag for your tests you can add a test_depend on rosbag and then you can know you can invoke it.

We added the extra capability to the doc jobs for two reasons. First is that they don't have a special doc_depend tag. And secondly because the doc jobs are not actually building it's much simpler to just make sure to check out the contents before running the documentation rather than requiring extra overlay logic. This is not something that users typically do and will work if they just have the packages in their workspace when they run the documentation for a package, if they don't it won't cross link but that's not a blocker.

[..] magic [..] magic [..] magic [..]

Aren't you being a bit dramatic?

I agree that things need to be reproducable, but really, being able to specify additional source dependencies for devel jobs specifically, configured to run on the ROS buildfarm is essentially similar to how people use Travis CI.

And additional source dependencies could very easily be managed using a .rosinstall or .repos file. That would again be following standard usage patterns of those files and I would assume would not be surprising or invisible/magic to users.

Users who are interested in running tests locally are probably also quite capable of following some instructions, and if those instructions include an additional vcstool import < my_unreleased_test_depends.rosinstall I doubt it will scare or confuse them.

In the end we are talking about a CI config. Not a generic "how to seed your workspace with my packages" workflow.

This is hyperbole.

The OP asked how to deal with unreleased test depends on the ROS buildfarm.

Not how to manage everyone's workspace.

Why is that unreleased code not in the same package/repo that's being tested?

you're assuming it's code. In the case of the OP, I happen to know we're talking about packages containing robot descriptions and similar resources.

That's not in the same repository as it doesn't belong there.

Why shouldn't it be released and versioned like any thing else

It is versioned, but in a git repository. It's not released because it doesn't make sense: it's not a "real" robot support package, but something crafted for tests specifically. Why "pollute" the ROS package namespace with a package which contains purely test artefacts?

This would lead to a situation like I have a workspace with A in it and everything is passing. But if I checkout B into the workspace and install it's test dependencies through some mechanism, then A's unit tests will start failing unless we

The OP asked how to deal with unreleased test depends on the ROS buildfarm. Not how to manage everyone's workspace.

My point is that if we open this code path up on the buildfarm it will significantly degrade the experience of users in workspaces.

Maintaining a .rosinstall is something that's completely reasonable for individual maintainers to do for their workflow for a single package or a handful of packages. But that customization needs to have a pipeline to propogate to the greater community and cover other use cases as well.

If you want to setup a custom travis job to test against unreleased code while you're developing something new and tightly coupled that's great and I support that. But the devel job are specifically scoped to show exactly what would happen if you take the code you have in the repository and build it against the targeted rosdistro as would happen if a user checked it out.

It's not released because it doesn't make sense: it's not a "real" robot support package, but something crafted for tests specifically. Why "pollute" the ROS package namespace with a package which contains purely test artifacts?

If it's a resource that you need to use to validate your package it's not "polluting", it's managing a resource to support your package and it's tests. Then it's real enough to exist. We have many packages designed to support other packages for testing. We restructure things all the time to support tests. If it's something that you want to be potentially shared by other packages as a generic test framework that sounds like any resource that should be released. If it's a oneoff that's only useful for this package it probably should be inside that package as it's not designed to be reused.

Thus if you're saying it something that deserves to be versioned separately then it probably deserves to be released separately so it can be relied upon being available and consistent.

There is the escape valve of checking out the content inside your tests but that still is fragile and requires network connectivity and the availability of git and other tools which would need to be added as test depends

To be clear we're well beyond the scope of the original question, but I'm communicating why not adding this "extra" functionality was a design decision, because it's asking the maintainers to do a little extra work (maintain and release your testing resources). However it will then significantly increase the meaningfulness of the test results. Any regressions are traceable to releases into the rosdistro. And testing resources are known not to conflict.

I appreciate the additional explanation.

It's likely we're not going to agree here, but that's not really a problem as it's not my question, but the OP's.

Your initial answer has already answered the question, and the answer is: you can't use unreleased test_depends on the buildfarm.