What's the actual difference between 文 and 语?
Yes, that is the main difference between 文 and 语. And you're right to say they refer more to written/spoken language rather than having that exclusive meaning - when someone talks about 中文, for example, they often mean the spoken language as well as just the written form, while 书面语 refers to literary words usually found in books. The other differences are explained by Fefe's post.

Here is a post (in Chinese) on Baidu Zhidao asking the same question, which comes to the same conclusion: http://zhidao.baidu.com/question/86849931.html