WPF async await TcpClient/TcpListener sample
The general issue with your code is that your client and server implementations reside in your control's code-behind. It is always a good idea to separate business logic from UI.
This is copy-paste:

if (anyIP.IsChecked == true)
{
    listener = new TcpListener(IPAddress.Any, Int32.Parse(serverPort.Text));
    Logger.Info("Ip Address : " + IPAddress.Any + " Port : " + serverPort.Text);
}
else
{
    listener = new TcpListener(IPAddress.Parse(serverIP.Text), Int32.Parse(serverPort.Text));
    Logger.Info("Ip Address : " + serverIP.Text + " Port : " + serverPort.Text);
}


can be refactored to:

var ip = anyIP.IsChecked == true ? IPAddress.Any : IPAddress.Parse(serverIP.Text);
var port = Int32.Parse(serverPort.Text);
listener = new TcpListener(ip, port);
Logger.Info(String.Format("Ip Address : {0} Port : {1}", ip, port));

Its a good style to prefix fields with underscore, so they can be distinguished from local variables. So it should be _listener instead of listener.
You should shut down your server on exit.
Yes, as long, as you await the call you can catch exceptions.