Atrial septal defect (ASD) is a common congenital anomaly that increases the risk of heart failure as well as strokes which can lead to cognitive impairment. The risk of stroke is higher when pulmonary hypertension develops and there is reversal of shunt. Stroke in ASD may be due to paradoxical emboli from the right heart or a left ventricular thrombus which develops as a result of atrial fibrillation, a common arrhythmia in ASD. We present a case of a 32-year-old Ghanaian man with history of ASD who presented with progressive memory loss with magnetic resonance imaging scan of the brain showing multiple infarcts, microvascular disease, and cerebral atrophy.