Target encoding with cross validation
Your question 1:

You are close but not exactly. 

Each time you take the mean, you include only the samples with the given value, say, A, of the categorical feature. 

You partition #2-#20 folds into 10 equal folds; however, once you consider only the samples with the category value A, their numbers may be different in these 10 folds. 

Say, these numbers are n1,...,n10. 

Then the mean of all but the first fold assigns the weight of 0 to the samples in the first fold and the weight of 1/(n2+...+n10) to each sample in all other folds.

The mean of all but the second fold assigns the weight of 0 to the samples in the second fold and the weight of 1/(n1+n3+n4+...+n10) to each sample in all other folds, etc.

Taking the average of these means yields the weighted average of the target with the weights of the samples being the averages of the above weights.

These weights will be slightly different for samples in different folds.

Thus, we get the weighted average of #2-#20 folds but with slightly different weights obtained by this random procedure.

You probably want to understand what is the point of this: how is it better than just taking the plain average over folds #2-#20?

This process adds randomality to the values of the encoding.  (The randomality can be measured, e.g., by the standard deviation of the encoding values for the category value A.)

This randomality of the encoding values may prevent the algorithm from learning the relationship between the encoding and the target in case this relationship holds only in the training set, i.e., reduce overfitting.

This randomality depends on two parameters:


The width of the distribution of the target value for the samples having the category value A.  (The width may be measured, e.g., by the standard deviation.)  Wider distribution in these target values results in greater randomality (wider distribution) of the encoding values.
The number of samples having the category value A: more samples result in smaller randomality of the encoding values.


One can compare this way with other ways of increasing the randomality of the encoding values.  The simplest way is to do single cross-validation but with less than 20 folds.  This will increase the randomality of the encoding values but it will mainly depend on the width of the distribution of the target values and less on the the number of samples in the category A.  This could be the reason for this double cross-validation.

Your question 2:

As you said, the point is to use for each sample the average of the target values that does not include this sample.

For a sample in the fold #1 with the category value A, the value of the mean target encoding is the (weighted) mean of the target values over all samples in the folds #2-#20 with the category value A (calculated as above).

This means that the values of the encoding of A will be different in different folds.

The post you quoted, does not mention the test set.  Other posts suggest that on the test set you take the average of the target values of all samples that have category A in the entire train set.