What information should be cached in experience replay for actor-critic?
The loss function is estimated in every batch training cycle. Gradients of the loss are computed and propagation back to the network happens in every cycle. This means that you use a small batch (e.g. 100 instances) from the replay memory, and by having the states you can input them to the respective network and have the $Q(s)$ for every state in your batch. Then you estimate the loss and run backpropagation and the networks' weights are getting updated. You continue gathering experience by interacting with the environment and after a threshold that you specify you repeat the cycle by re-sampling a new batch from the memory.

Just a suggestion: Start moving towards Asynchronous/synchronous methods for RL and use one network with different "heads" for Actor and Critic. Then you use one loss function plus the experience that now is collected from multiple instances of your agent-environment interaction (in contrast to your description in which one agent collects and stores experience from one instance).