While AtomGPT enables scalar material properties to be generated from atomic structures, its application for generating atomic structures from experimental properties, such as XRD, has not yet been explored. Based on these developments, we introduce DiffractGPT (DGPT) , a specialized generative model designed to directly predict crystal structures from powder X-ray diffraction (PXRD) patterns. DiffractGPT leverages the powerful architecture of AtomGPT, adapting it to the unique challenges of PXRD-based crystal structure determination. By training on large datasets such as JARVIS-DFT (JDFT), which comprises simulated PXRD patterns alongside their corresponding atomic structures, DiffractGPT learns to map complex diffraction data to accurate crystal structures. This approach enables the direct prediction of atomic arrangements from diffraction data, significantly reducing the need for iterative fitting and manual intervention. We further evaluate various application scenarios for DiffractGPT, such as XRD with no known chemical constituents, with guessed elements, and with explicit chemical formulas for structure design tasks. We also provide a web framework and tools to match the XRD patterns with existing data, as well as to generate new structures using the generative models. Most importantly, although we apply the models to XRD data, they can also be useful for other experiments, such as neutron and electron diffraction and other spectroscopic experiments.