Dimensionality reduction with prior knowledge of colinearity between features Let's say that I have sparse feature vectors and I'd like to use dimensionality reduction in order to visualize them more easily.

Dimensionality reduction techniques like PCA will estimate colinearity between the features given the data. What if I have some prior knowledge on the colinearity between my features? As in, I would be able to create an approximate distance matrix between my features, and therefore between my data points.

I am aware that if features are actually colinear, methods like PCA will find a way to reduce them, however, I am afraid that I don't have enough data points to infer that colinearity strictly from the data and that PCA would remove meaningful features and keep uninformative ones.

Let's say that my data looks like something like this:

$x_0 = \{1,NaN,-1,NaN\}$

$x_1 = \{1,NaN,NaN,-1\}$

In my case, I know that the 3rd and 4th features are colinear and that the distance between $x_0$ and $x_1$ is close to zero. Therefore, they should be mapped to a very similar data point in a lower-dimensional space.

Can PCA still do it with so few data points? Are there ways to "force" the known colinearity or distance measures? T-SNE perhaps?
One option is to use Multidimensional scaling (MDS) for dimensionality reduction. MDS can create a visualization of the relative positions for data based on the distance between the data points.

In your example, the data points that have no distance between them in several dimensions will be projected close to each other.