High udpclient CPU use
Note that Endpoints.Contains(endpoint) will never be true, because you create a new IPEndPoint, and it does not implement IEquatable interface.

In addition to @mjolka's description why you have a 50% CPU load, I would like to suggest how you can avoid it.

Currently you have a central place that manages all sockets in one place. I suggest to do it differently - have each endpoint handle the data on its own, and manage all sockets only as a collection of ongoing processing (tasks) like this:

public async Task RunListenerAsync(int port)
{
    var endpoint = new IPEndPoint(IPAddress.Any, port);
    using (var client = new UdpClient(endpoint))
    {
        while (true)
        {
            try
            {
                var result = await client.ReceiveAsync().ConfigureAwait(false);
                ProcessEndpointData(client, result);
            }
            catch (Exception exc)
            {
                logger.Error("Error receiving from channel: " + exc.Message, exc);
                return;
            }
        }
    }
}

private void ProcessEndpointData(UdpClient client, UdpReceiveResult result)
{
    var localEndPoint = (IPEndPoint)client.Client.LocalEndPoint;
    var device = (from d in Ports
                    where d.Key == localEndPoint.Port
                    select d.Value).FirstOrDefault();

    var data = new UdpData
    {
        Client = client,
        Data = result.Buffer,
        LocalPort = localEndPoint.Port,
        LocalIP = localEndPoint.Address,
        RemoteEndpoint = result.RemoteEndPoint,
        Device = device
    };
    Flow.bufferBlock.Post(data);
}


Note that in this implementation the only way to stop listening the port is to get an exception on await client.ReceiveAsync(). You might want to introduce the CancellationToken to have a control when to stop this process.

With this approach registering and listening for all ports will be as simple as:

public Task ListenPortsAsync(IEnumerable&lt;int&gt; ports)
{
    return Task.WhenAll(ports.Select(RunListenerAsync));
}