Why do classic psychedelics not cause withdrawal, despite high tolerance?
Short answer
Withdrawal effects are generally associated with drugs that induce feelings of euphoria by stimulating the reward center of the brain, either directly (cocaine, morphine) or indirectly (alcohol, nicotine). The classic hallucinogens have little effect on the dopaminergic pathways and hence do not cause serious withdrawal effects as observed with other classes of psychoactive drugs. 

Background
Withdrawal is defined by the WHO as: 


  a group of symptoms [that] occur on cessation or reduction of use of a
  psychoactive substance that has been taken repeatedly, usually for a
  prolonged period and/ or in high doses.


The typical drugs that are associated with withdrawal are alcohol and other sedatives, opioids, and stimulants.

Alcohol and sedative withdrawal can lead to anxiety, agitation, and depression. Opioid withdrawal results in drug-seeking behavior that may continue after the physical symptoms have abated. Stimulant withdrawal (the ''crash") is associated with depression, malaise, inertia, and instability (source: WHO).

All these characteristic symptoms are basically caused by the fact that the nervous system counteracts the drugs effects and downregulates the receptor systems activated by the drug. Therefore, more and more of the drug is needed. Indeed, downregulation is one of the physilogical mechanisms underlying tolerance (Stewart, 1993): 


  Tolerance refers to the decreased effectiveness of a given drug with repeated administration.


There are many factors underlying tolerance. For example, there is acute, short-term tolerance seen with cocaine use, but also a long-term tolerance form example seen in opioids. One factor in developing (long-term) tolerance is a downregulation of the receptor systems involved. Alcohol and other sedatives target the GABAA system. GABA being the primary inhibitory neurotransmitter in the central nervous system, it's not surprising that abstinence in a situation of tolerance results in over-stimulation of the nervous system (anxiety, agitation). Likewise, opioid tolerance results in mu-opioid receptor downregulation (Stafford et al., 2011). Given that mu receptors mediate positive reinforcement following direct (e.g.., morphine and other opioids), or indirect (e.g., alcohol, cannabinoids, nicotine) activation it is readily apparent why opioid abstinence results in the craving for the drug, as it is thought that after prolonged use, the user is unable to experience pleasure without the drug. At this point, the person is addicted (source: The Treatment Center). According to the NIH 


  Addiction is defined as a chronic, relapsing brain disease that is characterized by compulsive drug seeking and use, despite harmful consequences. 


Now we come to the conclusion that dopamine is central to addiction, also for drugs not directly activating it, such as alcohol and nicotine. The latter group indirectly release dopamine in the brain's reward center (the limbic system).

The classic hallucinogens stimulate the serotinergic system, and specifically the 5HT2A receptors (López-Giménez, González-Maeso, 2017), but fail to activate the reward centers in the brain. Indeed, all addictive drugs activate the reward circuitry of the brain, thereby producing the subjective “high” that the drug abuser seeks (Gardner, 2011). Tolerance to these drugs has rarely been reported (Smith et al., 1999) and LSD  rarely produces serious withdrawal symptoms (source: LSD Abuse Help. But in rare cases, tolerance can develop and is associated with 5HT2A receptor downregulation (Gresch et al., 2005) and the development of hallucinogen persisting perception disorder (HPPD). An LSD tolerance can be developed quickly, although it usually dissipates within just a few days depending on the user, their dosages and frequency of use. Long term tolerances are uncommon because users do not frequently repeat doses of LSD without taking breaks in between (source: Hallucinogens).

The SSRIs you mention basically work by downregulation of presynaptic 5HT1A receptors (Celeda et al., 2004), which are unrelated to the effects of hallucinogens.   

References
- Celeda et al., J Psychiatry Neurosci (2004); 29(4): 252–65
- Gardner, Adv Psychosom Med (2011); 30: 22–60
- Gresch et al., Neuropsychopharmacology (2005); 30: 1693–1702
- López-Giménez, González-Maeso, Curr Top Behav Neurosci (in press)
- Smith et al., Psychopharmacology (1999); 144(3): 248–54
- Stafford et al., Pharmacol Biochem Behav (2001); 69(1-2):233-7
- Stewart &amp; Baidani, Behav Pharmacol (1993); 4(4): 289-312