Yaghi was born in Amman, Jordan in 1965, to a refugee family originally from Mandatory Palestine. He grew up in a household with many children, had limited access to clean water and without electricity. 
At the age of 15, he moved to the United States at the encouragement of his father.  Although he knew little English, he began classes at Hudson Valley Community College, and later transferred to the University at Albany, SUNY to finish his college degree.  He began his graduate studies at University of Illinois, Urbana-Champaign and received his PhD in 1990 under the guidance of Walter G. Klemperer. He was a National Science Foundation Postdoctoral Fellow at Harvard University (1990–1992) with Professor Richard H. Holm. In 2021, Yaghi was granted Saudi citizenship.