LPF and HPF Confusion - Rectangular and Circular Cut-Offs
You're probably best off asking your professor for clarification. As you reasoned, it sounds like you're being asked to multiply the image in the frequency domain by either a rectangular or circular mask that eliminates all frequencies outside the mask. This is not typically how you would apply a filter, however, as "ideal" frequency-domain filter masks have unsatisfactory time-domain properties (e.g. a truly ideal rectangular filter has an infinitely long impulse response).

As you noted, filters with a tapered response (of which the Butterworth filter is an example) are more often used. However, the problem specification doesn't seem to contain the information that you would need to design such a filter, such as the type of filter and its order.