Use of progmem in ESP8266 vs AVR and also how to handle large dynamic strings This is a somewhat tricky question dealing with implementation and best-practices of the ESP8266. I'd ideally hope that it's answered by people with in-depth knowledge on the problem.

As you are hopefully aware, the Arduino (AVR) platform requires the use of the PROGMEM or F() instructions to avoid copying all strings to RAM. The ESP8266 platform also provides a PROGMEM instruction, but is memory-mapped.

I thus have several questions about what the PROGMEM instruction actually does on the ESP8266.

For the next few questions, until otherwise marked, I am referring to character-array-style C strings, not the Arduino String type.


Storage of constant strings: If I were to declare a string (global, for now) on the ESP8266, would it be read from flash only when it's accessed, or will it always take up RAM (like it would on the AVR. Assume that in this case I do NOT declare it as PROGMEM; I wish to find out what the default behavior is on the ESP8266)?
Local Constant Strings: What happens with strings that are used inside a function? I am fairly sure that they ALSO exist in RAM, at least in AVR. Does the ESP8266 do this too, or does it (potentially) store it in flash until it is read (it is fine if it ends up inside a cache somewhere, as long as it can be evicted if needed)?
Use of String(F("mystring")): If I wrap a PROGMEM string in this manner, does this truly convert the string into a RAM-based string only when this line is reached, or does this for some reason initialize the String() early (and consume the RAM anyway)? Do the systems differ in this behavior?
If I define several instances of the above (as in, the identical wrapped string shown above appears several times, perhaps in different functions), do all of them point to the SAME F() instance (since they are identical) or does it get duplicated since it's part of code rather than a referenced variable? On ESP8266? On AVR?
On the ESP8266, I discovered that the server.on() function for the built-in webserver read F()'d constant strings just fine, but failed to work once the file grew beyond the size of the RAM. It was clearly stored correctly, but it could not be read. I thus conjecture that the method was making a local copy to RAM, even if the file was stored in Flash. There was also a on_P version that worked after that point (I believe I used PSTR rather than F at this point as well, since -- correct me if I am wrong -- the two are not compatible). Is this the real point of PROGMEM on ESP8266? That the data is only stored in Flash, but it also tells the function reading it to make as few copies as possible or chunk it?




I am also curious what happens with large strings that aren't constant. The web server can return data that was passed via querystring, for instance, which in my case can be quite large. If I copy that data to a string (and then into a function), does it pass the heap pointer directly, or does each String make a new copy? I am concerned about fragmentation or other fun ways of running out of RAM even if one copy should in theory fit.

Does doing something like bool myFunction(String &amp;str){...} help? Some sort of way to actually ensure that it's the same string, rather than copying (and can I do String &amp;myStr=server.arg(...) -- is a return value different from an assignment)? It's confusing for me, since strings are clearly passed by value, despite being a complex datatype with dynamic memory somewhere in there. So, did they override the assignment operator (and/or the one that handles reference passing, since I think there's more than one) to duplicate the string as pass-by-value of a basic type would normally do, or do they share pointers (and thus memory) as a class passed-by-value would normally do?
On the ESP8266 flash is a very special case. Because the ESP8266 itself does not have any flash, but instead uses an external SPI-connected flash chip, accessing it is not an easy or technically obvious operation.

Reading from the flash chip is slow. Very very slow (by comparison to reading RAM).  Because of this where possible certain things are copied into RAM. Also, there are different sections of RAM that are dedicated to different purposes.


dram0 - This is your "Data" RAM. It's where your variables and such are stored (80kB total).
iram1 - Instruction RAM. It's here that interrupt routines need to be stored for rapid execution (32kB).
icache - Instruction cache. 16kB of data that is cached when reading from the flash chip. Executed code usually gets copied here.


Normally when you execute code the block of code that the instructions you are executing are copied from flash into the icache memory block and then executed. This is reasonably fast (reasonably in that there may be brief delays while copying the data from flash, but fast when actually executing. I don't know how efficient the cache is, whether it's split into different blocks that are loaded in the background by the memory controller as needed, but it seems to be a pretty efficient system that is used in practice).

Because of this disassociation of the CPU and flash string constants, unless specified otherwise, will be copied into dram0 during startup. This makes it very fast and simple to get at the data. Anything that you don't specify as PROGMEM will get copied into dram0. Note that you don't get all 80kB of the RAM to play with since the SDK uses a portion of it. You only get about 50kB at most (station mode - softap mode will use more). 

Now... what does PROGMEM do on the ESP8266 then?  Well, it adds an attribute to the definition that tells the linker to place the data in a specific section in flash (.irom.text.&lt;filename&gt;.&lt;line&gt;.&lt;number&gt;). This prevents it being copied into dram0 at startup.

So how do you get at it? That's where the *_P variants of the string functions (and certain overloaded functions in the Arduino core) come into play.

The whole of the flash chip is mapped (or can be, depending on register settings) into the ESP8266's memory map (address 0x40200000). Which means that it's possible to get directly at the data. However since this access doesn't go through the icache RAM block it's slow. Very very slow. About 12x slower by some accounts. Also because data can be in different locations within the chip the addresses of your string data can't be directly hard coded. There has to be some form of "it's over there" redirection when reading the data (for example if you use OTA updates you get two distinct blocks of flash where your code can be stored - the currently executing code and the previously executing version). The *_P variants of the string functions take that into account and look up where in flash the current code is located before reading from it. 

So to summarise: Not using PROGMEM gives you fast access to your string data at the cost of RAM, and limits the size of the data you can use. Using PROGMEM removes the size restriction, doesn't use RAM, but is considerably slower.

To specifically answer your questions:


The string data gets copied to RAM at startup.
The string data gets copied to RAM at startup.
The string data gets loaded from flash (using *_P variants of functions) when the String object is constructed (RAM is allocated on the heap at that moment in time). The RAM is freed when the String object is destroyed (goes out of scope).
The compiler has recently had an update that will deduplicate constant PROGMEM strings.
As mentioned above *_P copies directly from flash and is slow, but has no size limitation.