Which modules should be listed as dependency of my module? If my module calls a function of another module, that other module should be listed as a dependency of my module.

Now my module only provides some field handlers for the Views module. On the one hand, my module is useless without the Views module. On the other hand, my module does not call any functions or methods provided by the Views module.

Should I add Views to the list of dependencies for that module? If yes, would that change if my module provides other functionality unrelated to views (like custom pages)?

Note that I used Views only as an example. The question applies to all optional modules, that define hooks.
The Drupal.org .info file documentation does not define a dependency in itself other than stating a module cannot be enabled if a dependency is not present.

I guess you have to decide for yourself what you would consider a dependency. Personally, if a user needs to use Views in order to use your module than you would want to make sure it is installed.

If you don't want to make a module a dependency because it only supports secondary functionality in your module, you can use module_exists() to make sure your module doesn't die if the function it is calling is not there.