In the context of the optimism around antiretroviral therapy (ART) as prevention of HIV/AIDS, addressing the barriers to long-term ART adherence is critical. This is particularly important given the tendency to individualise or use a blame discourse when exploring why HIV-infected patients fail to adequately adhere to ART, and not sufficiently exploring contextual reasons for poor adherence that may require varying solutions. This study took place at three clinics and one hospital in Khayelitsha, South Africa, to document the contextual factors that challenged ART adherence in this community. Interviews were conducted with 20 HIV-infected patients who had defaulted on their ART and were subsequently admitted to Khayelitsha hospital for clinical complications, and 9 ART service providers including doctors, nurses and HIV counsellors. Interviews assessed the reasons patients defaulted on ART and explored ways this could be prevented. Data from both groups were analysed collectively using thematic analysis. While the interviews revealed a landscape of environmental risks threatening adherence to ART, all patients managed to overcome the identified barriers at some point in their treatment phase, indicating the fluidity of patients' needs and decision making. Patients reported that distrustful relationships with service providers could inhibit their understanding of ART and/or interrupt their follow-up at clinics. Patients described their rationale and agency underlying non-adherence, such as testing their bodies' physical limits without ART medication. The study speaks to the need to appreciate contextual social and structural barriers related to ART adherence, and how these are negotiated differently by specific sub-groups, to support an appropriate response. It is imperative to not solely emphasise loss to follow-up but also assess patients' subjective trajectory of their ART journey, decision making and agency with adhering to ART, their relations with healthcare workers, and how these dynamics are intertwined with broader constraints in health systems.