How to calculate camera distance so that after projective transformation a quad is filling the whole screen First of all, I'm using OpenGL and GLM. 

Now, I have a square perpendicular to the z-axis and centered around it. Let's say that it has Z-value (offset along the Z-axis) X. Now, I want to set up a projective camera, its position is (0, 0, Y) looking at (0, 0, 0). The perspective is: viewing angle=45 degrees, far=100, near = 0.1, aspect=1) I'm using glm::lookat() and glm::perspective(). 
How can I calculate the value for Y from X (given the parameters I've described)?
Fortunately your scenario is rather simple, your camera is on a line directly perpendicular above and centered on the quad you're concerned about. So what you want is for the quad to fill the whole screen, I assume (more or less) exactly the whole screen, i.e. the size of your projected quad in window space is your entire viewport.

The mathematics of this are rather easy if you understand how the perspective matrix is actually constructed and what it does and if you know some basic trigonometry. The matrix constructed with glm::perspective takes as viewing angle the field of view along the y-axis and is based on goold old gluPerspective. And if we take a look at the actual matrix, we can see that it transforms a $1$ high object sitting on the view axis at distance $1$ from the camera to a height of $\cot \frac{\alpha}{2}$ ($\alpha$ being your view angle). If you move it further away it obviously gets smaller and if you increase its height, it gets larger, so it's $\frac{h}{d}\cot\frac{\alpha}{2}$ for camera distance $d$ and object height $h$. We want that height to be equal to $1$ (the height, from the center, of the view plane in NDC space).

So all you need to know in addition to that is the size of your square in the y-direction. We take half that height (because the square is centered) and multiply that by $\cot\frac{\alpha}{2}$ to get the distance it has to be from the camera. So if your square vertices have the y-coordinate +/-H and given your other values:

$$Y-X = H\cot22.5°$$
$$Y = X+H\cot22.5°$$
$$Y \approx X+2.41421H$$