Colcon Skip Building Tests with test_depend Tag

edit

I've looked at https://answers.ros.org/question/1155... but my question is different.

We have a package A which depends on package B only during testing. When I use "test_depends" tag around package B in my package.xml for package A, it is always built. I cannot find a way to enforce that colcon does not build tests. Note here that -DBUILD_TESTING=OFF does not work as this is only enforcing things at the individual package level, not at the colcon level. How do I get colcon to not build package B when package B is specified in package A with a "test_depends" tag?