Recommended way to Implement One Time Password?
I do not fully understand the scheme you are describing, especially the interactions between your apps A and B. I'll try to describe a sensible HSM usage for a  OTP based authentication.

I think you have a little misconception about how a typical OTP is usually handled. There is no need to store an OTP ever, it is always calculated upon use. The typical scenario is, that the user (his OTP calculating app) and the server both store a shared secret. From that secret and some additional information (like a timestamp if you use TOTP, or a counter if you use HOTP) the one time password is calculated using the HMAC algorithm. 

This HMAC calculation can be done in software (if you use a normal app on your phone, like FreeOTP) or in hardware (like a PKCS#11 token) on both sides. The server side in your case, could use your HSM which stores the user's secrets to do the HMAC calculation. For a real massive multi user setting, you will probably have to create the shared secrets on the HSM dynamically from a master secret to overcome storage restrictions.

The specific procedure how you calculate your 6 digit (or whichever length you need) OTP ist outlined in the RFCs for HOTP and TOTP.