Is the following crypto compatible so that I can use the same keys for both products (Wireguard & Libsodium)? I am wondering if it is possible to use the public/private keys of two WireGuard peers also in another communication channel using libsodium?

According to the WireGuard Protocol documentation and the WireGuard paper, they are using:


  
  Curve25519 for ECDH
  ChaCha20 for symmetric encryption, authenticated with Poly1305, using
  RFC7539's AEAD construction
  


On the other hand, libsodium (which also supports a ChaCha20Poly1305 variant of the same construction) says:


  Sealed boxes leverage the crypto_box construction (X25519,
  XSalsa20-Poly1305)


Importantly, both are using DJB's Curve25519 but I am not sure if their keys are compatible, e.g. if they are using Weierstrass' or Edwards' notation. Can I use the WireGuard keys inside of libsodium, too? Do the two use compatible crypto?
WireGuard uses keys that are on the Montgomery curve, not the Edwards curve. It's the standard X25519 or Curve25519 commonly used for ECDH.

In libsodium, crypto_box_keypair calls crypto_box_curve25519xsalsa20poly1305_keypair which calls randombytes_buf(sk, 32), return crypto_scalarmult_curve25519_base(pk, sk). In WireGuard, wg genkey calls get_random_bytes(sk, 32) and wg pubkey calls curve25519_generate_public(pk, sk). It turns out that crypto_scalarmult_curve25519_base(pk, sk) and curve25519_generate_public(pk, sk) are essentially the same thing.

So this means in theory those keys would be "compatible".

However, you should not do this under any circumstances.

Cryptographic protocols make various assumptions on how keys are used. The WireGuard Protocol is formally verified, which means within the WireGuard handshake, it's been verified that none of the constructions will lead to any catastrophic situations. However, there's no guarantee that re-using these same keys within other random constructions that you might cook up won't interact badly and will retain the same guarantees of the WireGuard Protocol. So, if you reuse WireGuard private keys in your own protocol, you do so at your own risk. That's the kind of thing you only do if you really, really know what you're doing and are certain that there won't be any bad interactions.

(Perhaps a more productive way forward would be to describe what you actually want to do with WireGuard key identities and your libsodium-based protocol. Then maybe folks can help you find a safe way of accomplishing this. What's the overall end goal of your project?)