How should I modify a contrib module? I have to modify a contrib module with a custom function. In my case I can't use hooks to extend it. 

Is there a best practice, for example renaming the module and placing in sites/default/modules?
If your addition is general and not specific to your site, you could create an issue and post a patch there.

And even if you have a site specific extension, maybe it can be solved in a way that just requires an additional hook to the contrib module itself which could be useful for other use cases as well. Then you could again create an issue for that.