In this study, we compare the performance of different LLMs -OpenAI's GPT-4 Turbo (hereafter abbreviated as GPT-4) 8 , Anthropic's Claude 3 Opus (Claude) 9 and Google's Gemini 1.5 Pro(Gemini) -in two tasks, (i) generating question-answer (Q&A) pairs, and (ii) the extraction of synthesis conditions of MOFs, from the scientific literature by subject matter experts. Building on the RetChemQA dataset , we provide a thorough quantitative and qualitative comparison of each model's performance identifying the key strengths, limitations, and specific challenges encountered when generating the datasets. We find that for extracting synthesis conditions from the literature, Claude is the most comprehensive and accurate LLMs, while for generating Q&A pairs datasets Gemini performance is the best. Furthermore, we report the particulars of where these LLMs fail, but also where they demonstrate "amazing" and unexpected results. These findings provide insights towards advancing the use of LLM powered AI agents in reticular chemistry.