How can I perform categorical encoding when the dataset is too large for memory?
You can read the file line by line (or block by block), assuming it is in a format where that can be done, and keep track of the unique values of the category. 

Can handle multiple fields in 1 pass of the data instead of the 1 field like the below code.

Of course the easiest way is to get more memory or move to a cluster. The line-by-line solution will not be the fastest.

In pseudocode

unique_values_in_field_1 = set()

file = open(input)
while not EOF
    line = readline(file) 
    # can also read a block of lines and adjust the code accordingly
    field_1 = line.field_1
    # make a function to handle multiple fields during the same pass of reading the file
    if field_1 not in unique_values_in_field_1:
       unique_values_in_field_1.add(field_1)

close(file)

# now have all of the unique values in this field. Do whatever is needed to transform them.