Is it properly to include a third-party php library via composer without creating drupal module for it?
Sure, you can do that on a project.

But you're probably going to write code somewhere to actually integrate that into Drupal and expose it somehow, and that will be in a module (or theme). But especially if that's a custom module that is just used for that site, you kind of have to add it to the project composer.json as composer doesn't include composer.json of moules that aren't actually added themself with composer (excluding solutions like merge plugin)