How to force my ros package not using some ROS-provided libs?

edit

Let say I created ros package my_package that depends on library (or package)ext_package version 1. I already installed ext_package as stand-alone on my machine and everything worked fine so far...

Now, I created another new_package but needs to use ext_package provided by ros to read data from topic; need to use bridge. However ROS only provides ext_package V2, which is not compatible with V1. Am getting compilation issues.

So I would have to rewrite completely my_package to accomodate V2.... which would be amazingly tedious.

Thus the question: how to edit CMake file such that my_package uses standalone ext_package locate at ~/PathToLib/ext_package/build?

I already tried something like,

unset(EXT_PACKAGE_CONFIG_PATH CACHE)
 unset(EXT_PACKAGE_DIR CACHE)

 message(STATUS  "...initial CMAKE_PREFIX_PATH:")
 message(STATUS ${CMAKE_PREFIX_PATH})

 set(TMP_PREFIX_PATH ${CMAKE_PREFIX_PATH})
 set(CMAKE_PREFIX_PATH "/home/${usrname}/soft/lib/ext_package/build/installed")

 message(STATUS  "... corrected CMAKE_PREFIX_PATH")
 message(STATUS ${CMAKE_PREFIX_PATH})

  SET (EXT_PACKAGE_ROOT "/home/${usrname}/soft/lib/ext_package/build/installed")
  SET (EXT_PACKAGE_INCLUDE_DIRS "/home/${usrname}/soft/lib/ext_package/build/installed/include")
  SET (EXT_PACKAGE_LIBRARY_DIR "/home/${usrname}/soft/lib/ext_package/build/installed/lib")
find_package(ext_package REQUIRED)

... but without success. Any idea please?

P.S. In my case exta_package is ViSP. I installed version 2.10 as standalone on my machine. Also, am using ROS Kinetic.