Do not request a billing address during checkout
It is quite hard to do indeed. One way to do this is by setting a dummy billing address and hide the address form.

You will need to do the following:


In a custom module, add a CommerceCheckoutPane plugin. The plugin class should extend \Drupal\commerce_payment\Plugin\Commerce\CheckoutPane\PaymentInformation.
Override the buildPaneForm() method, in there:


Set an #after_build callback. In this callback, make each address field non-required. The reason that you need an #after_build callback is that else address fields won't be available.
On the commerce_profile_select element, set an #element_validate callback. In this callback, set the address values. The reason that you need an #element_validate is because a general form validation callback is called too late in the validation process to set the address in time. This is because the address element has a constraint and the address is validated against this constraint after #element_validate and before a general form validate. Should you try to set the address in a general form validation callback, then you'll get address field required errors because of the constraint.

You can set the #element_validate callback as well in the #after_build callback, which I've done in the code example below.

To hide the address form, you'll need to set #access to FALSE for the address form. This needs to happen in the #after_build callback as well, because earlier the address form isn't available yet.
On the checkout flow settings, disable the default Payment information pane (and enable the custom Payment information pane if not yet enabled).
To hide the dummy billing address on the 'Review' step, copy the template commerce-payment-method--credit-card.html.twig (and/or commerce-payment-method.html.twig if you are using a non-credit card payment method) to your theme and remove the line {{ payment_method.billing_profile }} from it.


Code example (file name: mymodule/src/Plugin/Commerce/CheckoutPane/PaymentInformation.php):


  

namespace Drupal\custom_commerce\Plugin\Commerce\CheckoutPane;

use Drupal\commerce_payment\Plugin\Commerce\CheckoutPane\PaymentInformation as PaymentInformationBase;
use Drupal\Core\Form\FormStateInterface;
use Drupal\Core\Render\Element;

/**
 * Provides the payment information pane.
 *
 * @CommerceCheckoutPane(
 *   id = "custom_commerce_payment_information",
 *   label = @Translation("Custom Payment information"),
 *   display_label = @Translation("Payment information"),
 *   default_step = "order_information",
 *   wrapper_element = "fieldset",
 * )
 */
class PaymentInformation extends PaymentInformationBase {

  /**
   * {@inheritdoc}
   */
  public function buildPaneForm(array $pane_form, FormStateInterface $form_state, array &amp;$complete_form) {
    $pane_form = parent::buildPaneForm($pane_form, $form_state, $complete_form);

    // Add an after build callback in order to make modifications on the address form.
    $pane_form['#after_build'][] = [$this, 'paneFormAfterBuild'];

    return $pane_form;
  }

  /**
   * After build callback for the pane form.
   */
  public function paneFormAfterBuild(array $pane_form, FormStateInterface $form_state) {
    // Get billing form element. Where it is located depends on the payment method that is chosen.
    if (isset($pane_form['add_payment_method']['billing_information'])) {
      $billing_form = &amp;$pane_form['add_payment_method']['billing_information'];
    }
    elseif (isset($pane_form['billing_information'])) {
      $billing_form = &amp;$pane_form['billing_information'];
    }
    else {
      // No billing information found.
      return $pane_form;
    }

    // Get the address form element.
    $address_form = &amp;$billing_form['address']['widget']['0']['address'];

    // Add element validation callback to autofill the address.
    $billing_form['#element_validate'] = array_merge(
      [[$this, 'profileSelectValidate']],
      \Drupal::service('element_info')-&gt;getInfoProperty('commerce_profile_select', '#element_validate', [])
    );

    // Set all address fields to non-required.
    foreach (Element::children($address_form) as $key) {
      $address_form[$key]['#required'] = FALSE;
    }

    // Hide the address form.
    $address_form['#access'] = FALSE;

    return $pane_form;
  }

  /**
   * Element validation callback for the profile select element.
   */
  public function profileSelectValidate(array &amp;$element, FormStateInterface $form_state) {
    // Set dummy address.
    $address = [
      'given_name' =&gt; 'A',
      'family_name' =&gt; 'A',
      'address_line1' =&gt; 'Dummy street',
      'postal_code' =&gt; '1234 AB',
      'locality' =&gt; 'Dummy city',
      'country_code' =&gt; 'NL',
    ];
    $form_state-&gt;setValue($element['address']['widget'][0]['address']['#parents'], $address);
  }

}