Where does signing/decrypting take place on a YubiKey 4/5 PIV private key? I have a YubiKey 4 and want to use it to store a private key that should never enter a computer system's memory. After importing the key to the device I understand it never leaves it.  Do the signing/decryption operations that take place for plain text using the private key happen on the device's hardware? Also, if this does happen on hardware, then can it be used like an HSM if proper compliance does not require that it generate the key on the device?
From the official specifications, published by NIST;
https://csrc.nist.gov/publications/detail/sp/800-73/4/final
NIST  Special  Publication  800-73-4 Interfaces  for  Personal  Identity Verification  –  Part  1:  PIV  Card Application Namespace,  Data Model  and  Representation

A.5 Key  Establishment  Schemes  with  the PIV  Key  Management  Key
FIPS  201  specifies a public key pair  and associated  X.509  Certificate for  Key Management. The  key management  key  (KMK)  is further  defined  in  SP  800-78, which  defines two  distinct  key establishment schemes for  the KMK:
A.5.1.1  RSA  Key  Transport  with the PIV  KMK
As specified  in  SP  800-78,  the on-card private KMK can  be an  RSA transport  key that complies  with [PKCS1].  In  the scenario  described  above,  a sender  encrypts a symmetric key  with  the KMK’s  public RSA transport key.  The  role  of  the  on-card  KMK private RSA transport  key  is to  decrypt  the sender’s symmetric  key on  behalf  of  the  cardholder  and provide  it  to the  client  application  cryptographic  module

The client application cryptographic module would typically mean the software on the computer that you connected the card to. It means that RSA operations happens on the card, but the computer will then decrypt the full message. It's designed this way to not make the card a bottleneck for large messages.
The Yubikey mostly follows the official spec for these protocols.