This is not possible. If by "writing tests" you mean this is something you only need for tests, a test_depend and an implementation of this feature request could help.
If the behavior is similar to ROS1, though, you could just omit to declare the dependency in the package.xml. This would still allow to compile the package the way you do. However, this obviously renders rosdep more or less useless, at least for that specific library package. You would need to manually ensure that the library package is installed, if you want to use it. And obviously, any releases via the ROS buildfarm will only work for the version without that package...