Maximum steganographic embedding rate not detectable by steganalysis? When using advanced steganographic methods (like F5) what is the approximate maximum embedding rate when advanced Steganalysis (like a machine learning approach) can no longer reliably detect it? 

I have searched for papers dealing with this topic but have been unable to find any. Does anyone here know anything about this topic or any research papers dealing with this issue?
It is an interesting question, let me clarify some points before. 

Nowadays F5 can not be considered an advanced steganographic method. There are algorithms as for example HUGO[1] or WOW[2] that are much harder to detect. Usually these algorithms hide information in the bitmap. F5 hide information in the DCT coefficients and this is easier to detect. Therefore, let's assume that we are using one of these modern algorithms.

Almost every research in modern steganalysis is based on machine learning. The common framework can be found in [3] and uses a feature extractor called Rich Models and Ensemble Classifiers. Using this framework (or reading this paper and other papers dealing with the same issues) you can find a first answer. An example result could be about 80% of accuracy using HUGO with bitrate 0.40. Maybe HUGO with a bitrate around 0.20 or 0.10 can be acceptable for you.

There are a lot of similar papers with different techniques and different results, getting better in recent papers. But there exist an important problem in steganalysis: the Cover Source Mismatch (CSM). It is, in fact, a problem of machine learning called "domain adaptation" with a new name [4,5]. It refers to the problem of using the classifier with images from a different source from those with which it has been trained. Unfortunately, we never known the source or domain of the steganographer, so we always have the CSM problem. And this worsens the results. 

If you take into account the CSM problem you will see that the results presented in some papers may be a little bit confusing. So, if you see something similar to my previous example (HUGO 0.40 with 80% of accuracy) you can not consider that HUGO 0.40 is insecure because if you have no access to the camera used by the steganographer, you can not train your classifier properly. Currently the CSM problems is an active area of research. 

At this point, reading the referenced papers, we have the optimistic answer (it will be for example 0.20/0.40 using HUGO) because the CSM problem and the cautious answer (it will be for example HUGO 0.10 or lower). 

[1] Pevný, T., Filler, T., Bas, P., 2010b. Using High-Dimensional Image Models to Perform Highly Undetectable Steganography. In: Information Hiding - 12th International Conference. pp. 161–177.

[2] Holub, V., Fridrich, J. J., 2012. Designing Steganographic Distortion Using Directional Filters. In: International Workshop on Information Forensics and Security (WIFS). IEEE, pp. 234–239.

[3] Fridrich, J., Kodovský, J., 2012. Rich Models for Steganalysis of Digital Images. IEEE Trans. Information Forensics and Security 7 (3), 868–882.

[4] Cancelli, G., Dorr, G., Barni, M., Cox, I. J., 2008. A Comparative Study of ±1 Steganalyzers. In: Multimedia Signal Processing, 2008 IEEE 10th Workshop on. pp. 791–796

[5] Ker, A. D., Bas, P., Böhme, R., Cogranne, R., Craver, S., Filler, T., Fridrich, J., Pevný, T., 2013. Moving Steganography and Steganalysis from the Laboratory into the Real World. In: Proceedings of the First ACM Workshop on Information Hiding and Multimedia Security. ACM, pp. 45–58.