Help with literature pointers for improving estimation of underlying mean for signal with periodic noise
You're looking at a cyclostationary random process. That's a concept you'll really need to understand! This will require a relatively solid basis on Fourier analysis, quite likely, but you might probably just take excourses into math basics literature while you read up on analyzing random processes and then specifically cyclostationary processes (really: do it in that order!).
So, Steven M. Kay: Fundamentals of Statistical Signal Processing: Estimation Theory is definitely a book you'd want to read, at least the first half, if this is the topic of your master thesis. That'd be the foundation of your &quot;state of the art&quot; chapter in your thesis, and it's really a good standard book that explains things in a very canonical way. Chances is your university library has it, or you can find it online for cheap (ask the library, and if that doesn't work, ask your professor to buy that book, by the way, if the university library doesn't have it. It's really, as far as I can tell, the standard book on the topic.)

with a specialization in machine learning

That means that you probably have a very solid foundation in stochastics, and my experience is that often, the relatively difficult jump from single (or families of) scalar random variables to stochastic processes has already been done, but often under some other name.

The naïve approach is to just take the average of the measured data, it’s clear that the main issue with this approach, is the effect of incomplete wavelengths.

Damn straight!

Of course if one could measure long series the periodic noise would average out, the goal of my thesis is rather to see how short it is possible to measure and still accurately estimate the underlying mean value

Good news: Well, I must admit this sounds either very unspecific, or very basic! In other words, soon as you refresh your stochastics knowledge and maybe extend it to a theoretical understanding of stochastic processes, you might be making relatively big jumps forward.
For example, assuming your noise $N(t)$ is additive, i.e. your observation $Y(t) = X(t)+N(t)$, and seeing that integration is a linear operation, you'll be pretty quickly be able to calculate the variance of your observation if your calculate the integral over some window $w(t)$, $\int_a^b Y(t)w(t)\,\mathrm dt=\int_a^b X(t)w(t)\,\mathrm dt+\int_a^b N(t)w(t)\,\mathrm dt$. Then, you'd throw a bit of the definition of cyclostationarity, and maybe Fourier analysis at the problem :)

One approach is tapering the edges of the signal, but we haven’t found any literature on that approach.

You must be reading the wrong literature! Thomson's Multitaper Method is indeed one of the classics in the analysis of random, periodic signals.

we will look into how to model the periodic noise as sine waves and then subtracting the periodic noise from the measured data.

You can never subtract noise unless you know it, that's the thing about noise, so you'll need to estimate it first, and that's a spectral estimation problem.