Unsupervised learning algorithms to detect anomaly in waves I have a sample of graphs (more than 10000...).
that look like in the image below:


I am searching for an unsupervised learning algorithms that can help me to detect anomalous observations.

Here what I suggest for beginning: for every observation I have a collection of points $(x,y)$. With this collection, I find Fourier series with regression (I compute coefficients with the base $\{1,\sin(x),\cos(x),\sin(2x),\cos(2x)\dots\}$).
Now I have a set of coefficients instead of waves.

Somebody have an idea how to detect anomaly?
Try Nonparametric Statistics. 
You may divide your random trajectories into parts and compare their statistic characteristics (mean and variance). Smth like this (but not exactly):

http://www.cscjournals.org/manuscript/Journals/SPIJ/volume7/Issue1/SPIJ-210.pdf