How did Shulgin take cognizance of the dosage of a new substance?
He didn't.
It was basically a matter of trial and error. Of course, having previous experience with other psychotropics of the same class helped him to have a rough estimate: in the case of MDMA, Shulgin was neither the first one to synthesize it nor the first one to use it, and he knew beforehand that the psychoactive dose was between 100 and 200 mg. Besides that, being a rational chemist, he started by what he supposed to be the smallest, safe dose.

Alexander Shulgin
For instance, regarding MDMA, Shulgin (2010) describes the following dosages (I'm copy/pasting only the start of each paragraph):

(with 100 mg) Beforehand, I was aware of a dull, uncaring tiredness that might have reflected too little sleep, and I took a modest level of MDMA to see if it might serve me as a stimulant...
(with 120 mg) I feel absolutely clean inside, and there is nothing but pure euphoria...
(with 100 mg of the &quot;R&quot; isomer) There were the slightest of effects noted at about an hour (a couple of paresthetic twinges) and then nothing at all...
(with 160 mg of the &quot;R&quot; isomer) A disturbance of baseline at about forty minutes and this lasts for about another hour...
(with 200 mg of the &quot;R&quot; isomer) A progression from an alert at thirty minutes to a soft and light intoxication that did not persist...
(with 60 mg of the &quot;S&quot; isomer) The effects began developing in a smooth, friendly way at about a half-hour....
(with 100 mg of the &quot;S&quot; isomer) I feel the onset is slower than with the racemate...
(with 120 mg of the &quot;S&quot; isomer) A rapid development, and both writing and typing are impossible before the end of the first hour...

Unfortunately, he doesn't explicitly say what was the sequence of the dosages he took... I can only assume it was the sequence of the paragraphs.
However, by far, the best example of how &quot;blind&quot; the chemist is regarding the adequate dosage of a unknown psychotropic is the famous case of Albert Hofmann, who invented LSD (not as a psychotropic) in 1938.

Albert Hofmann
When he decided to test LSD, in 1943, he ingested 250 micrograms... well, that's almost ten times a typical &quot;street&quot; dose of LSD nowadays, which is around 20 micrograms. Luckily for him, overdose of LSD (up to more than 100 times the amount he used) is not lethal.

Source: Shulgin, A. and Shulgin, A. (2010). Pihkal: A Chemical Love Story. Berkeley: Transform.