You can limit test-result to the package you are interested in: colcon test-result --test-result-base build/<pkgname>
You could also enumerate all found test result files with colcon test-result --result-files-only and use some shell commands to delete them. (Currently there is no command line option to do so for you.)