High IF Journal From Another Field vs. Low IF Journal In My Field
Let me suggest that the IF of the journal overall is less important to you than having your paper seen by the people who need to see it. I assume you are interested in citations of your work. You won't get them if the people in your field don't have ready access to your paper. 

I think that a specific journal is a (valuable) benefit to you, independent of IF, which is an overall measure that doesn't really impact any particular paper. 

This is especially true for hard-copy journals that have wide subscription within your field. Your paper will naturally appear before suitable readers. But even for online publication, your readers need to have ready access to at least the index of the journal in order to know it exists. 

The above is opinion, obviously.