Question about complement of frequency? (Using 过 and 次)
No, you can't.

The first sentence: 游 is a verb. 泳 is a noun.

The second sentence: 看 is a verb. 見 is an adverb. It is not the same structure as the first one.

You should just say 我看過你三次 or 我見過你三次.