Best buffering practice for continuous SD card writing
Is this a sensible solution to the problem?

No, unfortunately not. The size of the buffer is limited by the available RAM on your chip. Jumping between multiple buffers or cores will not change that, since it is a hardware limitation.

I am hesitant to implement anything that simply relies on an improvement in SD write speed, as I will likely need to increase the size of the data payload in the future and don't want to run into this issue again later on.

When you are continuously receiving faster than you can write, you only have 2 options:

Improve write speed
Reduce the needed data size

There is no other way, and that's a logical conclusion, not so much a programmical or hardware conclusion. It that situation the amount of &quot;buffered&quot; data will increase indefinitely, so you would need an infinite amount of memory. And that doesn't exist in real world. The same principle would apply to a high end PC with lots of memory, it will just be full after longer time, than with an ESP.

Improving write speed:

On various sites I have read, that you can use the SD card with different SPI frequencies. A higher frequency would mean a higher transaction speed. You can investigate further at this point.
Writing a full 512 byte page at once and only closing and opening the file again, if really needed, will improve the overall speed.


Reduce the needed data size

Maybe you can compromise with your data resolution and then pack those values as dense as possible (like packing two 4-bit values into a single byte of transmission).
You can research for compression code, so that you can compress your data lossless before transmitting it. Though the result will always depend on the kind of your data. Also mind, that in this case you are trading transmission size for computation time and RAM.



About buffers:
I would say, a normal circular buffer does not bring you much advantages here, since with the SD card you don't want to do bytewise data transmission. The 2 buffer approach, that you mentioned, seems good to me. Though doing handling with different cores does not make much sense to me. The UART receiving itself is done by hardware and some ISRs. You can do all the SD card transmissions in your main code, just making sure, that you regularly copy the received data from the Serial buffer into your buffer.