Why does azithromycin not affect human mitochondria?
There are two general points that should be appreciated in relation to this question:


Your statement that mitochondria “have prokaryotic ribosomes” is a misleading simplification. Although mitochondria and plastids are thought to be derived from eubacteria — and their ribosomes have some similarities  in antibiotic sensitivity — the structures of their large rRNAs differ (as indeed do those of their small rRNAs and their ribosomal proteins). This is important because the large rRNA (the equivalent of 23S rRNA in E.coli) is the target of many antibiotics, including the macrolide antibiotics, of which azithromycin is a member.
Experiments in vitro have shown that human mitochondrial ribosomes are not sensitive to some of the macrolide antibiotics that inhibit eubacterial ribosomes, whereas those of some lower eukaryotes are.


Therefore it should not be particularly surprising that macrolide antibiotics do not affect human mitochondria, or that they do affect Toxiplasma etc., the ribosomes in the apicoplasts of which are sensitive to these antibiotics in vitro — presumably because of their rRNA structure in the specific target region.

An answer to the question in your title would seem to require an explanation of the molecular basis of these different sensitivities. It is thought (although not, I gather, definitely proved) that this is the identity of the base at the position equivalent to 2058 of E.coli 23S rRNA: those ribosomes that are sensitive to the antibiotics have the base A in that position, whereas those that are resistant have the base G.

References

Böttger, E.C. et al.(2001) EMBO Reports vol 21, pp. 318–323. Structural basis for selectivity and toxicity of ribosomal antibiotics.

Greber, B.J. et al.(2014) Nature vol 515, pp. 283–286. The complete structure of the large subunit of the mammalian mitochondrial ribosome.