Determining input color space primaries from spectrum data If I have an output device, for which I have a known spectrum data for each of it's primaries, I should be able to calculate the CIEXYZ of its primaries using using the integral equations together with the CIE color matching functions. I can then calculate its respective CIExyY values, which for output device are always inside the boundary if the xyY spectral locus - the chromaticity diagram.
My question relates to calculating xyY of primaries of input device.
Suppose I have a known spectrum data for the primaries of some input device - be it a digital scanner or a digital camera or any other input device. It's color filters are not perfect, this means that even a pure spectrum color is represented as non-zero values in its own RGB color space. That means that (if I'm correct) that the whole chromaticity diagram should lie inside of the device RGB color space, and it's primaries lie outside of the diagram and be represented by imaginary colors.
How do I calculate those imaginary primaries for the input device given known spectrum for its primaries? If I'm not mistaken, I cannot simply use the integral formula, because that seems to be only for output devices and the resulting primaries would lie in the locus.
I vaguely understand your question, so this response should be taken more of a long comment rather than an answer.

If I have an output device, for which I have a known spectrum data for each of it's primaries, I should be able to calculate the CIEXYZ of its primaries using using the integral equations together with the CIE color matching functions. I can then calculate its respective CIExyY values, which for output device are always inside the boundary if the xyY spectral locus - the chromaticity diagram.

This is correct in general but note that reference CIE spectral power distribtions are defined for a standard observer.
The term input primaries is a little vague, are you refering to the spd of the light source inside the input device or the combined result produced by the object's spectral reflectance factor (also an spd) multiplied by the spd of the light source of the input device ?
If you know the spectral reflectance factor of the object and the color matching function and the resulting trichromatic value, you should be able to express the spd of the light source as a vector equation something like:
$x_1 \begin{matrix}a_1 \\ a_2 \\ \vdots \\ a_{\lambda}\end{matrix}+x_2 \begin{matrix}a_1 \\ a_2 \\ \vdots \\ a_{\lambda}\end{matrix}+...+x_{\lambda} \begin{matrix}a_1 \\ a_2 \\ \vdots \\ a_{\lambda}\end{matrix} = \begin{matrix}b_1 \\ b_2 \\ \vdots \\ b_{\lambda}\end{matrix}$ where $x$ is from your light source spd and the others are obtained from the remaining (you can for example distribute trichromatic value uniformly to $b$ vector, and fill $a$'s with zeros for non corresponding wave lengths and use the multiplied value for the corresponding wave length), but this is more or less an educated guess rather than a tested solution.
Notice that the expression is a linear system which might help you finding a solution.