Are retroviruses cytotoxic?
As you point out human beings (and other organisms) have hundreds of retroviruses and many can be seen in the human genome, some of which appear to be the remains of deactivated viruses, others which may be active.   Most retroviruses are not pathogenic - they don't cause disease.  

This is because many retroviruses replicate slowly, budding and secreting from the cell surface in small numbers without killing their host cell. 

There are several disease-causing retroviruses.  Human T-cell Leukemia Virus (HTLV) can cause T-cells to replicate in an uncontrolled fashion, resulting in leukemia.  This is not a necessary result for HTLV to replicate, but it is a result of viral infection as viral replication tweaks the cell machinery to replicate. 

While it's not clear exactly how, Human Immunodeficiency Virus (HIV) kills its hosts cells.  Unfortunately HIV's hosts are CD4 immune cells.  The fact that sporadic replication of HIV can cause all host immune cells to die entirely makes HIV a pathogen. 

Disease is usually thought of as a lack of adaptation of a pathogen and host: viruses do better if they do not kill their host.  Both these cases are possibly not the future for either virus: either could adapt to reproduce without causing a disease. Or they might have already , having spun off some strains into the pool of quiet retroviruses medical science does not concern itself with.