Is there a way in MATLAB to approximate an FIR filter using an IIR Filter?
Sure there is.

The simplest version would be to calculate the transfer function of your FIR filter, sample it on some meaningful frequency grid and then shove it into invfreqz() to design the IIR filter.

Depening on you FIR filter this may work well or be a total disaster. In my experience invfreqz() does often very poorly. 

If that's not good enough you can certainly write your own search an optimization algorithm. However, this is NOT a beginner project and requires significant math and coding skills.

See mathworks.com/help/signal/ref/invfreqz.html#mw_3c79f471-99ad-46f8-a680-14c29ff47d07