How can I include a third party library in my custom module without using Composer Manager
General: Composer Merge Plugin

In your Drupal's main composer.json you can use the Composer Merge Plugin to scan certain directories for composer.json files and merge them into your main composer.json.

You'd then have a composer.json in your custom module requiring the 3rd party library. In your root composer.json you require the merge plugin and configure it under the extra section to include certain directories.

{
    "require": {
        "wikimedia/composer-merge-plugin": "dev-master"
    },
    "extra": {
        "merge-plugin": {
            "include": [
                "web/modules/custom/*/composer.json"
            ]
        }
    }
}


Optional: Composer Custom Type Installer

If the 3rd party library in your custom module is available on GitHub and you want to define a location where the lib should to be downloaded to (extra helpful especially for all front-end libs) you can use Composer Custom Type Installer in your main composer.json.

You'd then require the 3rd party library in the repositories section in your custom module's composer.json assigning it a package type you pre-configured in the extra section of your main composer.json to define a location where the lib should be downloaded to.

"repositories": [
    {
        "type": "package",
        "package": {
            "name": "select2",
            "version": "4.0.3",
            "type": "theme-library",
            "source": {
                "url": "https://github.com/select2/select2.git",
                "type": "git",
                "reference": "tags/4.0.3"
            }
        }
    }
]


And the corresponding extra section in the main composer.json:

"extra": {
    "custom-installer": {
        "theme-library": "web/themes/custom/MYTHEME/libs/{$name}"
    }
}




Of course, for simplicity you could also do everything (composer-merge-plugin and custom-installer and requiring the GitHub repo) just from you main composer.json. I'd recommend that.

If you do so, then run at least once composer require PACKAGENAME (in my example composer require select2) after you manually incorporated the lib into the composer.json file.



Last but not least, don't forget to bind your library with MYMODULE.libraries.yml and hook_page_attachments (latter called from MYMODULE.module). See Adding stylesheets (CSS) and JavaScript (JS) to a Drupal 8 module.