LM338 Voltage and Power Regulation
Never assume.

Read the data sheet.
R2 and R1 set the voltage - and that ONLY in the diagram you posted.

There is an equation in the data sheet (on the same page you took the diagram from) that explains how to calculate the output voltage from R2 and R1.  The only tricky part is Iadj, which you have to find elsewhere in the datasheet. 

The LM338 is a voltage regulator.  You can use an LM338 as a current regulator, but that is a different circuit from the one you gave.  If you need to regulate voltage and current, then you need to use the circuits given later in the datasheet.

You can only leave out the capacitors if the power conenctions are less than 6 inches (15cm.)  In any case, you are better off including them.  They don't cost much, and are much cheaper than replacing the hairs you ripped out while searching for a problem that could have been avoided by including them (the capacitors.)

The datasheet mentions the 6 inch limit for whether or not you need capacitors.  It doesn't say anything about the voltage playing into that decision, so whatever advice you got isn't backed by the datasheet.

As for the poor regulation:
You must cool the LM338 - it will need a good heatsink when operating at 5A.  If it gets hot, it will shut down.

The LM338 needs a good 3Volt difference between input and output voltage to work properly.  Your 9V input is too close to the required 6V output to work correctly and reliably.