How to make a very large lookup table in Verilog?
Your first example errors because you must use a variable data type on the left hand side in a procedural block.
xml is the left hand side of the assignment, and so must be a variable (reg,integer,etc.), not a net (wire). Outputs default to being wire unless otherwise specified. You can fix this with an output reg [2:0] xml in your port declaration list.
Alternatively, given you are basically describing a ROM, you can use an initial block to set the default values for your lookup table:
localparam SIZE = 1024;
reg [2:0] rom [SIZE-1:0];

initial begin
    $readmemb(&quot;rominit.txt&quot;, rom);
end

assign xm1 = rom[x];

Alternatively, given you are using Quartus, you could use a MIF file:
localparam SIZE = 1024;
(* ram_init_file = &quot;rominit.mif&quot; *) reg [2:0] rom [SIZE-1:0];
assign xm1 = rom[x];


Finally, and perhaps more simply, your mapping can easily be described algorithmically:
module gray_decrement #(
    parameter WIDTH = 3
)(
    input  [WIDTH-1:0] x, 
    output [WIDTH-1:0] xm1
);

genvar i;
generate

// First convert to binary
wire [WIDTH-1:0] bin;
assign bin[WIDTH-1] = x[WIDTH-1];
for (i = 1; i &lt; WIDTH; i = i + 1) begin : gray2bin
    localparam idx = WIDTH-1-i;
    assign bin[idx] = x[idx] ^ bin[idx+1];
end

// Then subtract one
wire [WIDTH-1:0] binm1;
assign binm1 = bin - {{{WIDTH-1}{1'b0}},1'b1};

// Then convert back to gray code
assign xm1[WIDTH-1] = binm1[WIDTH-1];
for (i = 1; i &lt; WIDTH; i = i + 1) begin : bin2gray
    localparam idx = WIDTH-1-i;
    assign xm1[idx] = binm1[idx] ^ binm1[idx+1];
end

endgenerate

endmodule

The above should optimise away to a lookup table which should decrement the gray code value.
(As a side note, your lookup table is wrong - the default value should be 4 not 0, otherwise a value of 0 maps to 0, and nothing maps to 4).