Clustering categorical variable values based on continuous target values
Since you are looking for a degree of similarity regarding $y$ and the values of $x_1,...,x_5$ do not matter you can view this as a clustering problem regarding $y$:

Let $y_1,...y_5$ be the target values with $f(x_i)=y_i$ for $i\in\{1,...5\}$ then you need to define a distance measure $d(y_i,y_j)$ which, since your variables $y$ are continuous, could be the euclidian distance: $d(y_i,y_j) = (y_i-y_j)^2$. But you could also choose the absolute difference. (note that I am assuming your $y_i$ to be one-dimensional here, i.e. $y_i\in \mathbb R$). That gives you a formula to measure "does not vary that much". 

It also provides an answer to your second question: you can choose from a range of unsupervised ML algorithms to do clustering. The most popular one probably being $K$-means. The idea here is very straight forward: 

1. select a number of clusters K
2. initialize the position of the K clusters randomly
3. Assign each y_i to the closest cluster
4. For each cluster k calculate the new cluster position as the mean of all y_i belonging to that cluster
5. repeat 3 and 4 until the assignments of y_i does not change anymore


Mathematically this gives you a mapping $c(y_i)=k$. 

And eventually when you are done with your clustering you just pick a cluster $k$ and for each $y_i$ in that cluster you look up the corresponding $x_i$. Which will return what you asked for.