xterm commands (ls, cd, ...) unable to access Desktop, Documents after updating to Catalina
The solution (at least for me) was to add the binary xquartz to the list of app with "Full Disk Access" (in the "Privacy" tab of the "Security and Privacy" setting of the "System Preferences"). The path to the binary is:

/opt/X11/bin/Xquartz

I also added 

/opt/X11/bin/xinit 

since it's a running process, although I don't think it is needed...

However, I am still having problems accessing iClouds folders (Document, Desktop, etc.)