How to avoid Msg 9514, Level 16, State 1, Line 3 error for linked server?
What is probably happening is that SQL Server wants to download the whole remote table to your tempdb before attempting to join to the local table, including the XML column. 

Try something like this:

SELECT TA2.ID 
FROM [DB].[dbo].[table1] TA2
INNER JOIN (
    SELECT col1, col2, col3, CAST(xmlCol AS varchar(max)) AS xmlCol
    FROM [linkedsrv].[DB].[dbo].[table1] 
) AS PA 
    ON (TA2.ID = PA.ID)


Or like this:

SELECT TA2.ID 
FROM [DB].[dbo].[table1] TA2
INNER JOIN (
    SELECT * 
    FROM OPENQUERY(linkedsrv, '
        SELECT col1, col2, col3, CAST(xmlCol AS varchar(max)) AS xmlCol 
        FROM [DB].[dbo].[table1]
    ')
) AS PA 
    ON (TA2.ID = PA.ID)