Meaning of Actor Output in Actor Critic Reinforcement Learning
When using the loss function for the critic described in your question, the Actor-Critic is an on-policy approach (as are most Actor-Critic methods). Your intuition as to what it is learning seems to be quite close, but the notation/terminology is not quite on point.

First it's important to realize that the $Q(s, a)$ critic is an estimator, we're training it to estimate state-action values. You could say that we are training it such that it can hopefully provide accurate estimates of:

$$Q_w^{\pi} (s_t, a_t) \approx \mathbb{E}_{\pi} \left[ r_t + \gamma V^{\pi}(s_{t+1}) \right].$$

You'll notice I've added quite a number of symbols there in comparison to your $Q(s_t, a_t) = r_t + V(s_{t+1})$:


I have added the $\pi$ superscript to $Q$ and $V$; this denotes the behaviour policy, which is the policy that we're using to generate experience. In on-policy methods, this is equal to the target policy (the policy for which we're learning to predict values). Adding this superscript makes explicit the fact that we're learning expected returns for states and state-action pairs that are only accurate under the assumption that we continue following the $\pi$ policy from state $s_{t+1}$ onwards.
I added the discount factor $\gamma$, which is probably just a tiny detail you forgot.
I added $\mathbb{E}_{\pi}$ to indicate that we're trying to estimate an expectation under $\pi$ (and the environment's dynamics).


So, the critic is trained to estimate $Q^{\pi}(s, a)$, which can intuitively be interpreted as the long-term discounted rewards that we expect to collect when executing $a$ in $s$, and selecting actions according to the distribution $\pi$ subsequently. It definitely still is trying to estimate $Q(s, a)$ values for state-action pairs, not just $V(s)$ values for states alone.




  What In actor critic outputs $Q(s_{t+1},a_{t+1})$?


In practice, when using the loss function described in your question, $a_{t+1}$ really simply is a single action selected in an actual trajectory of experience by the policy $\pi$. The trained network simply takes $s_{t+1}$ as input, and the output corresponding to a single action $a_{t+1}$ as selected by the policy is used as the value for $Q(s_{t+1},a_{t+1})$ in the update rule. 

The update rule does not involve any sum over all actions, multiplied with their probabilities. The "trick" is that we do not just run the update rule a single time, but we expect to generate lots (sometimes millions) of trajectories as experience, and we repeatedly run the update rule. In different trajectories, we'll experience the different actions $a_t$ as samples with approximately the correct frequencies, and in expectation we'll have proper update targets (except for potential bias resulting from function approximation).




  I want to use actor critic with experience replay in an environment with a large action space


The Actor-Critic method in your question is, as I mentioned above, on-policy. This means that the experience used in update rules has to be generated according to exactly the same policy for which you are also learning value estimates. This is incompatible with the idea of experience replay, because old trajectories stored in a replay buffer were generated by older versions of your policy.

There are off-policy Actor-Critic methods which can correct for the mismatch in distributions and use experience replay, but these are going to be quite a bit more complicated. Examples are ACER and IMPALA.