 Mortgage refinance costs swung up due to a new fee, which some are calling a tax. The announcements by Fannie Mae and Freddie Mac say the lump-sum fee of one-half a percent of the loan balance applies to refinances on single family homes. 