Where do I place helper files? I would like to place some (PHP)functions in a helper files, which requires across modules and themes.

Do I have to create a new module or I can place a simple helper file with list of common functions, what is the best practice for these kind of functions in drupal 8 ?
Yes, create a utility module, so your consumer modules can declare a dependency to the utility module in their .info.yml file. Optionally to keep the admin UI clean you could use hidden: true in the utility module's .info.yml

(I assumed you are speaking of a very drupal-specific helper lib, not some general purpose library like e.g. Guzzle that would go into composer)