HMAC key storage for a web app? If one is going to use an HMAC, I.E: http://php.net/manual/en/function.hash-hmac.php, where exactly are they storing the key?

Without some sort of secure storage (i.e not in a local file or code), I feel like one is fooling oneself when using a HMAC (as with the above function). I missing something related to the storage of keys?
Before we get into the key storage and management issues, I wanted to clarify the scheme that is proposed in the linked article, to make sure it is clear what we are talking about.
Let's assume we get pw for authentication from the user and retrieve salt and tag from the database and we get sk from the storage method (see below) being a secret key. We then only accept the password pw if and only if tag=hmac(sk,salt||pw||salt) holds. The prototype of hmac looks as follows: MessageAuthenticationCode = hmac(SecretKey,Message), where the input SecretKey is implicit for trusted devices which provide a parameter for designating a specific secret key stored inside the device.

I really only see three options you have for storing the key if you want to use to do the password verification on a server that reboots:

Use a trusted device connected to the server. This includes smart cards (they may be too slow), trusted platform modules (TPMs; which may also be too slow) and hardware security modules (HSMs; which may be too expensive)
Store the encrypted key and hold the clear one in RAM and require user interaction for unlock. This may work by sending the server a secret password that allows for decryption of the HMAC key if it's at rest or may use this protocol (with EC-ElGamal) for smart card / TPM / HSM based / software based remote unlock.
Store the encrypted key temporarily, hold the clear one in RAM and use a trusted device. This means you let the server generate the secret HMAC key which is only held in RAM until a reboot happens and use a trusted device on reboot only. You may combine this with the second approach.

Now for the details and trade-offs of each method.
Trusted Devices
You can get yourself a smart card and install it in the server. You then send the password to be checked to the card, which holds the key, and receive the corresponding password hash which you compare to the stored value. This may not be suitable for mass-authentication as smart cards tend to be slow  but is the (second) cheapest option among the trusted devices.
You can get a server mainboard featuring a trusted platform module (TPM) and use it like the smart card in the above paragraph. However the TPM suffers from the same problems: While it tends to be cheap it usually is rather slow and not suited for a larger service (handling more than ~10 requests per second).
The &quot;best&quot; solution from this category is the hardware security module(HSM). It would be used just like the other two. Whatsoever those modules tend to be very fast and would very-well be capable of handling many requests in very short times. Furthermore HSMs tend to also have better security certifications then smart cards and TPMs. The downside of course is that HSMs are very expensive, with the cheap ones starting at several hundred dollars and the better ones going up to several ten-thousand dollars, whereas smart cards are within the price range of a few dollars to a hundred dollars per piece.
TL;DR: So overall, while they provide the most security, I don't think trusted devices would solve your situation, because of API, speed and price restrictions.
User Interaction
Here there are basically four different options. A combination of in-place or remote password-based or public-key based key unlock. I'll quickly describe each.
But first I need to describe the overall concept of this approach. You let the server generate the HMAC key in RAM and store an encrypted copy on the hard drives. Every time the server reboots, the HMAC key gets wiped from the RAM and the encrypted copy needs to be unlocked in order to verify passwords. This is where the user interaction comes into play.
The first version of this would to simply enter the password while being directly and physically connected to the server. This is arguably the most secure solution of those requiring user interaction but on the other hand also the most time-intensive solution, as you have to plug-in a monitor and a keyboard, every time your server restarts, so you won't want to do this.
The second version is nearly identical to the first, but this time you'd connect a smart card reader (with a pinpad) to the server, enter the PIN and let the card (asymmetrically) decrypt the stored HMAC key. While this is even more secure than the first approach it suffers from the same problems and thereby usually isn't an option.
The third version improves upon the first, by establishing a TLS (or otherwise) secured channel to the server and handing the password over this way. While this is obviously less secure than the first version as there's the risk of intercaption and man-in-the-middle attacks it is also a lot more scalable.
The fourth version improves upon the second. You'd establish a secured channel (TLS, IPSec, local network, whatever) to the server and then run this protocol on it. This will allow you to decrypt the stored key without the key (or the ciphertext) ever having to leave the server. Yet only the correct smart card / private key can unlock the HMAC key. This is the most secure and most scalable solution among the remote-solutions that require user interaction at boot.
TL;DR: This may be an appraoch that is suitable for your needs, but may require too much user interaction to scale well. And of course the security of a fully hardware based solution can't be reached which may be acceptable as the API, speed and price can be met.
Trusted devices on Reboot
Full disclosure: I didn't invent this approach, but observed it first at Jetico's BestCrypt VolumeEncryption.
You can let the server generate the key in RAM. Now you let the key stay there until a scheduled reboot happens where the TPM will encrypt the HMAC-key and store it onto the drive. Now the server reboots and after the reboot the TPM will unlock the key and afterwards delete it. This allows for unattended reboots but has the problem that if an unexpected reboot happens the key will be lost and no passwords can be verified. The positive side is of course that this will scale extremely well and will allow for fast software based verification while using powerful hardware protection when necessary.
TL;DR: While this solution will scale quite well, it has the issue of key-backup in case the server shuts-down without a warning. And of course the security of a fully hardware based solution can't be reached which may be acceptable as the API, speed and price can be met.
The optimal solution (?)
This approach will work the same as the previous one with the small exception that one makes an encrypted backup of the HMAC-key to the drives. Most of the time the server will operate as in the above approach (using software-based HMAC for verification), but in the case of reboots the TPM will kick in and temporarily make the back-up copy for the reboot. Now the weakness of the &quot;old&quot; approaches was that there's no backup for the case of unexpected reboots. In such cases the user interaction appraoch could kick in and allow for decryption of the back-up key. This would even fix the problems of the &quot;blind-decryption&quot; protocol as the user now would very rarely use this method and only in cases where it was sure what happened, allowing for stronger key protection (like a smart card in a vault) and where the user can directly see the result (successful unlock). Furthermore the backup copy of the HMAC-key could be moved to a physically secure location (like a CD inside a vault) which was one of the problems of the stand-alone protocol.
TL;DR: This approach fixes the issues with the previous one, so there's a solid back-up, allowing for unexpected reboots and for unattended planned reboots. And of course the security of a fully hardware based solution can't be reached which may be acceptable as the API, speed and price can be met.
Note: Drives may be directly attached to the processor / mainboard or may be at a physically different location.