Canonical explanation of Android Encryption and Vulnerabilities Note: Well, the bounty expired and one possible reason could be effort required to address as I gather from comments. Seeing the number of upvotes, it appears to be of interest to others as well. I would still like to get an answer so here's what I propose- a good answer within a month, will get a bonus of 50. This would I hope give adequate time and incentive



I have been trying to understand Android Encryption process and its vulnerabilities for a while

There are many questions addressing portions of this topic on this site and also on sister site.  To drive my point home, these questions address portions and not the whole (reminiscent of "blind men and an elephant ?" :)


What is Marshmallow encryption protecting?
Does full device encryption protect my data from Google and the government?
https://android.stackexchange.com/questions/137334/if-we-encrypt-our-device-can-user-connecting-to-adb-get-it
Questions pertaining to SD card encryption
android-encryption-when-phone-is-on
android-cyanogenmod-encryption-vs-gnu
is-there-cryptographic-material-in-a-phones-sim-card-that-can-be-used-with-rsa
android-device-encryption



My understanding (or misunderstanding ?)


Encryption password is generated from a combination of user lock screen PIN and encryption algorithm (therein lies an inherent weakness owing to limited length of PIN)
This is salted, and stored in root location, not accessible to users
This is used to  generate the actual password to encrypt /decrypt and the actual password is stored in RAM
This was strengthened by linking Step 1 to the device SoC (Which Android Version ? Which is the hardware element that uniquely identifies the device?  Can that be replaced to fake?)
Hence, it is not possible to decrypt data without encryption key and device (holds for external SD as well)
Possible recovery methods - brute force, capturing RAM information (step 3) to obtain key
Rooted devices appear to be more susceptible to access step 2 data through custom recovery/ possibly ROM and kernel flashing?? (if true, why is this not touted as a big risk?)
Even if this information is obtained , I am guessing that it is non-trivial effort wise to generate the actual password 
Marshmallow can treat external SD as "internal storage" or "portable storage". Logically, it shouldn't make a difference but am not sure


There are gaps in my understanding, probably missing out on other key aspects as well.

So, I am looking for a canonical explanation for  understanding from a user perspective


Entire encryption process (including external SD)
Implementation variation across Android versions- from KitKat to Marshmallow (including dual options for external SD in Marshmallow)
Vulnerabilities at a user level 


Note


I am aware of the risk of the question being considered too broad but IMO warrants  a comprehensive treatment
Having some experience in communication security, I understand the challenge in translating Cryptographic concepts to a user level. I would prefer the answer to address this, with explanatory pointers for deeper understanding. Examples of the process need not be Cryptographically correct in a rigorous sense but should convey the essence
A possible advantage could be "duping" future questions on related aspects
At the cost of repetition, answers should be primarily at user level, but with adequate explanation for a deeper understanding. Splitting the answer in two portions may be a suitable way.
I would make it a point to down vote trivial/casual/patch work answers to encourage comprehensive answers
I envision it works like this:


Storage is encrypted using synchronous random key.
When the user chooses or changes a password that is based on whatever input, be it a password comprised of letters and numbers and characters, or be it a pin code, or pattern swipe, or finger print, or any other input, an asynchronous encryption algorithm is used to encrypt the master key, such that correct identification ends up decrypting the input resulting in the master key, which in turn makes it possible to encrypt and decrypt storage.
The moment the user logs out, the memory holding the master key is overwritten


The big trick here is that asynchronous encrypting of the master key. Once Android has the master key, it has the ability to exchange data with the storage. Only when the user is logged in, that master key is known. Asynchronous encryption is what is called public key encryption. What happens is that a public key encrypts data (the master key in this case), and a private key decrypts data. Not to be confused with the storage encryption here. The storage is just synchronous encryption. There the same key is used to encrypt and decrypt. But the finding/retrieval of that "master" key, is the biggie. It means that if at one point you have a weak login method, like for intance "1234" as a pincode, and you change your mind, and change the pincode to "5364", which is harder to guess, unless that earlier "1234" was stolen, snooped on, at any point, the security has just become better. Same deal when changing the login method to a full password that is impossible to guess or dictionary attack. The storage itself does not need to be re-encrypted at all. It's all about hiding that master key - internally. The user never sees that master key, because it's most probably just a random hash code of sorts - nothing will ever "find" or "guess" that hash code. Not even the NSA or any other security body on the planet could ever find a matching key like that.
The only attack vector is hoping for a weakness on the user's part. Perhaps the user chose a pincode login. If it's 4 digits, then it's a maximum of 10000 possible pincodes. The OS might "block" the device after trying out a few in a short time. The solution is then to "hack" the OS, so that it becomes possible to try all possible pincodes without the OS intervening and blocking the device. I believe that is how the FBI eventually got access to a phone of a criminal. A third party company (some Israeli company from what I recall) did the hacking for the FBI I think. They bypassed that pincode-try limit.
If the login is a full password, and if the user picked a strong password, and you're sol. Not in a life time with all the cpu power on the planet will hack that in a million years. I'm not buying any of the NSA can decrypt anything rumors. I think those people watched one too many men-in-black movies. All one has to do is look at the scientific documents about the various encryption algorithms (eg. AES), and you'll know that hacking simply won't happen, except in the olden days when there were 40 bit keys. Those days are long gone. AES128 is already unhackable I think, and if anyone is concerned, jumping to AES256 makes it more secure by a magnitude to the size of the universe pretty much.
Maybe one day quantum computers might decrypt it, but I'm skeptical. Not sure if it is possible to have a probability system simply highlight the solution. We'll see about that, eventually. Perhaps that's a few lifetimes away anyway. Nothing to worry about right now.

So, end of day, the security limitation lies entirely on the login method that is used. One can change the method without having to re-encrypt the storage. All this because of the asynchronous public key encryption of the master key.