You probably wont even have THAT loan long enough for you to recoup the money that you paid upfront! It would take more than 5 years for you to recoup the difference. 90% of homeowners refinance within the first 5 years. What I see mostly of, however, is 2 to 3 years! At least 1% lower would be good to refi to.