I couldn't find a fix to the error, I ended up switching to moveit::planning_interface::PlanningSceneInterface::applyCollisionObjects which seems to mostly work without errors so far.