Recommended way to Implement One Time Password? I need help with the process/commands needed to implement the below functionality:

I'm modeling an OTP System, in which the user would request an OTP and the app will send him one. Later the user will use the OTP to use some functionality or something, and the OTP needs to be validated.
Constraints: 


The OTP cannot be stored in the Clear.
Need to use Thales payShield 9000, that I have available, sending HOST COMMANDS.


This is the process:


App A request an OTP using a User Account
App B Generates the OTP for that User Account and send it to App A
App B Can't store the OTP in clear, so it must use the HSM to, in some way, store the OTP that can be validated later
App A, later, sends the OTP to App B which validates it (using the HSM) and then procedes to send the approval to App A


My apologies if this doesn't make sense, and I don't have idea what I'm doing.
I do not fully understand the scheme you are describing, especially the interactions between your apps A and B. I'll try to describe a sensible HSM usage for a  OTP based authentication.

I think you have a little misconception about how a typical OTP is usually handled. There is no need to store an OTP ever, it is always calculated upon use. The typical scenario is, that the user (his OTP calculating app) and the server both store a shared secret. From that secret and some additional information (like a timestamp if you use TOTP, or a counter if you use HOTP) the one time password is calculated using the HMAC algorithm. 

This HMAC calculation can be done in software (if you use a normal app on your phone, like FreeOTP) or in hardware (like a PKCS#11 token) on both sides. The server side in your case, could use your HSM which stores the user's secrets to do the HMAC calculation. For a real massive multi user setting, you will probably have to create the shared secrets on the HSM dynamically from a master secret to overcome storage restrictions.

The specific procedure how you calculate your 6 digit (or whichever length you need) OTP ist outlined in the RFCs for HOTP and TOTP.