Frankly, if you considered the proposition out of the question you would have politely put him off long ago.
Frankly, if you considered the proposition not remotely possible you would have politely put him off long ago.