What kind of "vector" is a feature vector in machine learning?
I'm having trouble understanding the use of Vector in machine learning
  to represent a group of features.


In short, I would say that "Features Vector" is just a convenient way to speak about a set of features. 


Indeed, for each label 'y' (to be predicted), you need a set of values 'X'. And a very convenient way of representing this is to put the values in a vector, such that when you consider multiple labels, you end up with a matrix, containing one row per label and one column per feature.

In an abstract way, you can definitely think of those vectors belonging to a multiple dimensions space, but (usually) not an Euclidean one. Hence all the math apply, only the interpretation differs !

Hope that helps you.