Instead of C:\SQLDBA\ServerName\Instance rather C:\SQLDBA\ServerName_Instance
Your script seems unduly complicated. All you need to do to format the folder name according to the title of your question is replace the backslash in @@SERVERNAME with an underscore. No need for the table, script file, and XML PATH for this task. PowerShell also includes a New-Item command to avoid the need for the legacy COM FileSystemObject.

DECLARE @PowershellCommand varchar(8000) = 'powershell -Command "New-Item -Path \"C:\SQLDBA\'
    + REPLACE(@@SERVERNAME, '\', '_')
    + '\" -ItemType \"directory\""';
EXEC sp_configure 'xp_cmdshell', 1;  --assumes 'show advanced' config option is already enabled 
RECONFIGURE;
EXEC xp_cmdshell @PowershellCommand;
EXEC sp_configure 'xp_cmdshell', 0;  
RECONFIGURE;


You could also avoid xp_cmdshell entirely and just run a stand-alone PowerShell script on the server:

New-Item -Path "C:\SQLDBA\ServerName_InstanceName" -ItemType "directory"