PCA and Eigenvectors
Each eigenvector of the covariance corresponds to a new feature (which is a combination of the original features).

The principal eigenvector of the covariance matrix corresponds to the direction of maximum variance (not least as mentioned in the question). 
Subsequent eigenvectors can be computed one by one under the constraint that they are orthogonal to the ones already computed. The fact that they are orthogonal makes the new features uncorrelated with each other.
In other words, a set of $k$ eigenvectors of the covariance corresponds to a set of $k$ new features that are uncorrelated with each other.

There are many different viewpoints on what exactly principal components are. In general, these seemingly different viewpoints have strong connections with each other. For example, the first $k$ eigenvectors (i.e., the eigenvectors corresponding to the $k$ largest eigenvalues of the covariance matrix) capture the maximum cumulative variance of the data. At the same time they form a basis for a $k$-dimensional subspace (the space of new features) such that if you project your original data (each datapoint individually) on that subspace (i.e., if you optimally approximate each datapoint using the new features), then the total euclidean distance between the original datapoints and their projections is the minimum possible; by minimum we mean in comparison to any other $k$-dimensional subspace.

This short article takes a look at what PCA is from a few different viewpoints.