Reading from SPIFFS - What's the best way to deal with a string (or char) array with undefined length? I've read that it's best to avoid using String in Arduino so I'm ready to use a char array if it's better.
I'm reading the file names from an SD Card and my goal is to store them in an array and then be able to read that array. What I'm currently doing is:

String lib_arr[100];
int lib_arr_size;


and then

File dir = SD.open("/");
int c = 0;
while (true) {
  File entry = dir.openNextFile();
  if (!entry) {
    // If no more files
    lib_arr_size = c - 1;
    break;
  }
  if(strcmp(entry.name(), "SYSTEM~1") == 0) {
    continue;
  }
  lib_arr[c] = entry.name();
  entry.close();
  c++;
}


and then

for (int i = 0; i &lt; lib_arr_size; i++) {
  if(i == pos) {
    tft.print("&gt; ");
    tft.println(lib_arr[i]);
  } else {
    tft.println(lib_arr[i]);
  }
}


This is not a solution because, for my purposes, the String lib_arr[100] can be 1000 or 10000 or 100000... and I found out that by increasing that value the global variables use skyrockets and Arduino says that it's out of memory. My goal is memory optimization. I don't know how to go about this problem.
Well, you cannot use more memory than you have. If the length of file names is (probably/possibly) too much, you just cannot store it in memory. Some alternatives:


Use some compression... This will save you a bit, but at the end if you don't know how much the maximum bytes you need to store is, it will not help much
Use external SRAM memory... You need to write your own memory functions. Also a slight performance hit.
Use SD card memory... As SRAM, but much slower. Also not sure if you can use the same SD card to write on.
The best solution, but could be a big performance hit, is not to store at all the file names, but go through the file names and process what you need along the way. This is not always possible or feasible though. In your solution you can print the file names directly without storing (but I guess you want to do some additional processing).