Selection of recombinant host by color selection method
The Z gene you are talking about should be lacZ that encodes the $\beta-$galactosidase, which is made from $\alpha$ and $\omega$ peptides. Neither peptide is functional by itself.   

$\beta-$galactosidase will cleave the glycosidic bond in X-gal and form galactose and 5-bromo-4-chloro-3-hydroxyindole. The latter product then dimerizes and oxidizes to 5,5'-dibromo-4,4'-dichloro-indigo, an intense blue product that is easy to identify and quantify.

Next, you should check out the genotype of your bacterial strain because they typically have only the $\omega$-peptide encoded in its genome. Thus, the enzymatic activity of the $\beta-$galactosidase is only recovered when the $\alpha$-peptide is in the environment, and the process is called $\alpha$-complementation.   

Two commonly used strains for cloning and expression in E. coli are Dh5$\alpha$ and TOP10.

The genotype for Dh5a is like this:  


  F– Φ80lacZΔM15 Δ(lacZYA-argF) U169 recA1 endA1 hsdR17 (rK–, mK+) phoA
  supE44 λ– thi-1 gyrA96 relA1


Here, lacZΔM15 indicates that the M15 segment is lost from lacZ, and the mutant protein will only have the $\omega$-peptide. 

The genotype for TOP10 is like this:


  F– mcrA Δ(mrr-hsdRMS-mcrBC) Φ80lacZΔM15 ΔlacX74 recA1 araD139 Δ(ara leu) 7697 galU galK rpsL (StrR) endA1 nupG


Again, you will notice lacZΔM15, which is still a non-functional peptide segment after all.

Finally, to your question. Even if the there is no glucose as primary energy source in the medium and only X-gal is present, the operon of your host bacteria(I suppose it's E. coli) cannot produce the functional $\beta-$galactosidase on its own. Moreover, the recombinants, with the vector sequence encoding the $\alpha$-peptide disrupted, will not exhibit $\alpha$-complementation. No color reaction then! :D

Reference:
1.Genotypes of Invitrogen™ competent cells - TOP10, retrieved from: https://www.thermofisher.com/cn/zh/home/life-science/cloning/competent-cells-for-transformation/chemically-competent/top10f-genotypes.html.
2.Genotypes of Invitrogen™ competent cells - DH5a, retrieved from: https://www.thermofisher.com/us/en/home/life-science/cloning/competent-cells-for-transformation/chemically-competent/dh5alpha-genotypes.html.
3. Langley, K. E.; Villarejo, M. R.; Fowler, A. V.; Zamenhof, P. J.; Zabin, I. (1975). "Molecular basis of beta-galactosidase alpha-complementation". Proceedings of the National Academy of Sciences of the United States of America. 72 (4): 1254–1257. doi:10.1073/pnas.72.4.1254. PMC 432510 Freely accessible. PMID 1093175.
4.