Make embedding more Gaussian-like
One easy approach is to use a modified VAE loss with a stochastic encoder. In other words, let $z=E(x)\in\mathbb{R}^n$ and $y = D(z)$ be your pseudo-decoder (not a decoder, since you said your output is not meant to perform reconstruction; in fact it's not strictly necessary at all). Then, as in VAEs, define the probabilistic encoder $E(x)$ via
$$
\mu(x) = f_{\theta_1}(x), \;\; \Sigma(x) = g_{\theta_2}(x), \;\; z\sim \mathcal{N}(\mu,\Sigma)
$$
where $\Sigma$ is a diagonal covariance matrix. Use the reparametrization trick for backprop. Commonly this is written $z\sim q_\theta(z|x)$.

Now, let your current loss (not necessarily reconstruction) be written $\mathcal{L}(x,z,y)$. Our "pseudo-VAE" regularized loss for one data point is then just: 
$$
\mathfrak{L}(x)=\mathcal{L}(x,z,y) + \beta\, \mathcal{L}_\text{KL}(x,z)
$$
where $\beta\in\mathbb{R}^+$ and the KL loss is written as
$$
\mathcal{L}_\text{KL}(x,z) = \frac{1}{2}\left[
-\log|\Sigma(x)| - n + \text{tr}(\Sigma) + \mu^T\mu
\right]
$$
assuming you want to be $\mathcal{N}(0,1)$ distributed.



Alternatively, you can treat each minibatch as approximating the marginal distribution, and then do moment matching. Concretely, let $X=(x_1,\ldots,x_n)$ be a minibatch of inputs, and $E(X)=Z=(z_1,\ldots,z_n)$ be the corresponding batch of encodings (embeddings). Suppose you want to make it "look like" $\mathcal{N}(\mu,\Sigma)$. Then let
$$ \widehat{\mu}(Z)=\frac{1}{n}\sum_i z_i \;\;\;\;\;\&amp;\;\;\;\;\; \widehat{\Sigma}(Z)  =\frac{1}{n-1}\sum_i (z_i - \widehat{\mu}(Z))(z_i - \widehat{\mu}(Z))^T $$
so that our moment matching penalty (which you can add to your current loss function) is just
$$
\mathcal{L}_\text{M}(X) = \alpha |\mu - \widehat{\mu}|^2 + \gamma || \Sigma - \widehat{\Sigma}(Z) ||_F^2
$$



Another, slightly more complex approach, is to use an adversarial method, as in adversarial auto-encoders, adversarially learned inference (ALI), etc...
Basically, you train another network $C(z)$, which you train like the discriminator of a GAN (e.g., LS-GAN, WGAN-GP), but your generator is just $E(x)$. You train your critic $C$ to differentiate $ z\sim\mathcal{N}(0,I_n) $ and $z=E(x)$, and maximize its loss when training the "generator". This is equivalent to a conditional GAN, where the noise input is fixed (a Dirac Delta) or non-existent, the output is just the embedding, and the true samples are simply normally distributed random vectors.