How to respond when rejected with encouragement to apply again, when hiring committee says position is always open?
This is indeed unusual.  For tenure-track positions at traditional universities in the US, a normal search proceeds on a specific timeline, from application deadline through interviews and leading up to the new hire's start date, usually the beginning of an academic year.  Once a position is filled, no further applications are accepted until there is a new opening and a sufficiently high administrator has given permission to begin a new search.

So in this normal situation, if you have been rejected, there is no point in reapplying until a new opening is posted, and typically that would be the next year at earliest.  (And there's no particular guarantee that the same department will have any openings at all the next year.)  So if you still want to apply to this employer, you have to wait a year; otherwise, consider it a loss and focus on other opportunities.  

Given all this, the only way that this reply makes sense to me is if the university has a policy that applications will technically always be accepted.  That way if an extremely strong candidate decides to apply out of the blue, even if no opening currently exists, the university can consider creating a brand-new position just for them.  (For a major research university, "extremely strong" probably means something like "Nobel laureate".)  If you were such a candidate, you'd probably know.  But otherwise, this is of no practical value, and I suggest you continue to follow my previous advice and consider it a loss.  (And if you've already been rejected in one search, without even reaching the final round, then it seems clear they don't consider you "extremely strong", sorry to say.)

It's also possible that whoever replied to you is just confused.  In that case, too, I think you should fall back to the above and consider it a loss.

In either case there is not much point in responding to the message at all.

Also note that any phrase like "we encourage you to apply next time the position is open" should not be taken literally.  It's a mere courtesy which they surely write to everybody who is rejected.  It does not necessarily indicate that they really want to hire you.

Emory's suggestion of "apply once a week" is unwise, in my view.  This would be very annoying to the search committee, and make you look rather crazy.  It  would probably torpedo all hope of ever getting hired by that department.