How are the toxic components in a mixture identified?
You have many questions nested together. Whenever you have a mixture + identification of components in your mind, bring chromatography to your mind. You must be familiar with paper chromatography. Same idea, but applied to more sophisticated instruments.
When you have a completely unknown mixture, you start a so-called screening procedure with a high-performance liquid chromatograph and a mass spectrometer. The HPLC separates the components, and the mass spectrometer identifies the molecules from the fragmentation pattern and molecular mass.
You start to see how many components are there in the HPLC-MS chromatogram. Let us say you had natural products and you see 100 peaks (which is not unusual at all).
Later you scale up the process, go at a preparative scale and start collecting fractions of mixtures. You perform biological tests on them, and the relevant fraction is further studied by more advanced tools such as recycling the HPLC-NMR system.
The question you ask is equivalent to a decent Ph.D. project in natural product isolation and characterization followed by investigating their biological activity. In pharmaceutical companies, HPLCs run day and night with a typical HPLC may have 10-15 columns for screening as to which column under which mobile phase will produce the best separation.
I did not check how fagopyrins were discovered, but the process is general. Wiki gave a hint on the first German reference Fagopyrin, ein photodynamischer Farbstoff aus Buchweizen (Fagopyrum esculentum) = Fagopyrin, a photodynamic dye out of buckwheat, locked behind paywall. This procedure is from 1950s, when HPLCs did not exist but the technique of chromatography was very well known.

Wir haben aus einem Methanolextrakt von getrockneten Buchweizenblüten
durch fraktionierte Verteilung und chromatographische Adsorption einen
photodynamisch wirksamen Farbstoff in Form eines dunkelroten, fast
schwarzen mikrokrystallinen Pulvers isoliert, der im folgenden als
Fagopyrin bezeichnet ist.
We have isolated from a methanol extract of dried buckwheat flowers by
fractional partitioning and chromatographic adsorption a
photodynamically active dye in the form of a dark red, almost black
microcrystalline powder, hereafter referred to as fagopyrin.