How did Mendeleev know elements from compounds or mixtures?
Please do not underestimate the scientists of 19th century. They were as creative, intelligent and perhaps more genuinely dedicated to science than the scientists of the 21st century. Spectroscopy was the tool of the trade to identify and verify that a given substance is not a mixture.
The original reference which established that Didymium was a mixture is in German [1] (English title: &quot;On the decomposition of didymium in its elements&quot;).
I don't know enough German to read it conveniently, but it is clear that the authors did fractional crystallization of a lot of salts, made measurements absorption spectra, did emission spectroscopy and finally led to a beautiful result. The original paper has plenty of spectra! From [1, p. 490]:

Da sonach die exacte Zerlegung des Didyms in mehrere Elemente realisirt ist, so schlage ich vor, die Bezeichnung Didym nunmehr ganz zu streichen und beantrage, für das erste Element, entsprechendder Grünfärbung seiner Salze und seiner Abstammung die Benennung:
Praseodym mit dem Zeichen $\ce{Pr}$
und für das zweite, als das „neue Didym“, die Benennung:
Neodym mit dem Zeichen $\ce{Nd}.$
Aus den Atomgewichtsbestimmungen, die in allen Einzelnheiten nach Bunsen ausgeführt wurden, ergaben sich als vorläufige  Werthe für Praseodym = 143·6, für Neodym = 140·8, wobei dem entsprechenden Oxyd die allgemeine Formel $\ce{M2O3}$ zukommt.

English translation with assistance of DeepL:

Since, therefore, the exact division of the didymium into several elements has been carried out, I propose that the name 'didymium' be deleted altogether and that the first element be given a name corresponding to the green colour of its salts and its origin:
Praseodymium with the sign $\ce{Pr}$
and for the second, as the &quot;new didymium&quot;, the naming:
Neodymium with the character $\ce{Nd}.$
From the atomic weight determinations, which were carried out in all individual units according to Bunsen, it resulted as provisional values for praseodymium = 143·6, for neodymium = 140·8, whereby the corresponding oxide is assigned the general formula $\ce{M2O3}.$

Anyway, historically, it is not Mendeleev who had to decide whether the elements were actually elements. You may ask the right question, how did the element discoverer decided that it was an element in Mendeleev's time? The answer is that each element has a story to tell and that is why Isaac Asimov wrote a beautiful book called &quot;The Search for the Elements&quot;. Go ahead and get hold of it.
Reference

v. Welsbach, C. A. Die Zerlegung des Didyms in seine Elemente. Monatshefte für Chemie 1885, 6 (1), 477–491. DOI: 10.1007/BF01554643.