How can I require a custom module hosted elsewhere with its contrib dependencies being downloaded too?
Just for this question I created a dummy module at https://github.com/leymannx/custom_module which has a composer.json similar to yours with drupal/restui in the &quot;require&quot; section and &quot;type&quot;: &quot;drupal-custom-module&quot;.
And then in a D8 project's composer.json I adjusted the &quot;repositories&quot; section to look like this:
&quot;repositories&quot;: [
    {
        &quot;type&quot;: &quot;composer&quot;,
        &quot;url&quot;: &quot;https://packages.drupal.org/8&quot;
    },
    {
        &quot;type&quot;: &quot;github&quot;,
        &quot;url&quot;:  &quot;https://github.com/leymannx/custom_module&quot;
    }
],

In the &quot;installer-paths&quot; section I added:
 &quot;web/modules/custom/{$name}&quot;: [&quot;type:drupal-custom-module&quot;]

I then simply ran composer require leymannx/custom_module and it downloaded the module and drupal/restui both to the right places. Of course you now can ignore the custom module with your repo's .gitignore file.
I guess the trick was to have the &quot;repositories&quot; section adjusted like above. And I guess you also can use proper versioning for the custom module when you tagged some releases. Right now it just downloads dev-main.