how to extend ctools functions in my own js file?
JavaScript doesn't allow extending a function with another one. If you want to extend modalContentClose() you just redefine it in your JavaScript file that is included after the JavaScript used by ctools.module.

modalContentClose = function() {
  // Add your own code.
  close();
  return false;
};


Your version of modalContentClose() cannot refer the previous modalContentClose(), if you don't save it in a new variable, as with the following code.

originalContentClose = modalContentClose;

modalContentClose = function() {
  // Add your own code.
  originalContentClose();
};