Fixed 5V regulator circuit LM1084
To used the 5V fixed should I connect Adj/GND pin directly to ground?


Correct - confirmed from section 5 of the datasheet:



where it says that pin 1 is: "Adjust pin for the adjustable output voltage version. Ground pin for the fixed output voltage versions."