Using a bathroom fan with a smaller duct - is there a minimum size?
The small duct will not make your fan &quot;useless&quot;.  However, the fan will not perform to its published air handling or noise specifications. Further, its lifetime may (or may not) be diminished and its warranty might be voided. Buy from a seller with a liberal return policy, try it, and return if you are not satisfied with the result.
Another consideration: You could replace the duct with the appropriate size for its entire length EXCEPT for the point where it penetrates the masonry. Although not ideal, this will be much better than with the undersized duct because of the duct's length.  Finally, performance will be even better if the duct is replaced with one that is oversized. This is probably good practice anyway due to its fairly extreme length and this will partially compensate for the flow restriction at the masonry penetration.