Create line breaks from text field
Expanding from Pascal Mollet's answer, this was the solution I came up with.

First setting the address variable with the |nl2br filter:

{% set address = contact.address|nl2br %}


Then, outputting that variable with the |raw filter:

{{address|raw}}


Using them both together at once like this (as Pascal suggested):

{% set address = contact.address|nl2br|raw %}


didn't work.