Is there a method that is opposite of dimensionality reduction? I am new to the field of machine learning, but have done my share of signal processing. Please let me know if this question has been mislabeled.

I have two dimensional data which is defined by at least three variables, with a highly non-linear model way too complicated to simulate.

I have had varying level of success at extracting the two main components from the data using methods like PCA and ICA (from the python library Scikit-Learn), but it seems these method (or at least, these implementation of the methods) are limited to extracting as many components as there are dimensions in the data, for example, 2 components from a 2D point cloud.

When plotting the data, it is clear to the trained eye that there are three different linear trends, the three color lines show the directions.



When using PCA, the main component is aligned to one of the color lines, and the other is at 90°, as expected. When using ICA, the first component is aligned with the blue line, and the second is somewhere in between the red and green ones. I am looking for a tool which could reproduce all three components in my signal.

EDIT, Additional info: I am here working in a small subset of a bigger phase plane. In this small subset, each input variables produce a linear change on the plane, but the direction and amplitude of this change is non-linear and depends on where exactly on the bigger plane I am working. At some places, two of the variables can be degenerate: they produce change in the same direction. for example, say the model depends on X, Y, and Z. A change in the variable X will produce a variation along the blue line; Y causes a variation along the green line; Z, along the red one.
The short answer is yes. 

Essentially you will be performing some sort of feature engineering. This means constructing a series of functions of your data, often:

$\phi_j(x): \mathbb R^p \rightarrow \mathbb R \ ,\ \  j=1,...,K$

Which, strung together, define a transformed data vector $\boldsymbol \phi(x)$ of length $K$.

There are a number of ways, better and worse, of doing this. You may want to look up terms like:


Splines and generalised additive models.
The kernel trick (how to make a model where $K\rightarrow \infty$).
Feature engineering (of the manual variety, e.g. adding an $x^2$ column to your data).
Deep learning, representation learning


As you might guess from such a varied bag of techniques, this is a large area. It goes without saying really but care has to be taken to avoid overfitting.

This paper Representation Learning: A Review and New Perspectives deals with some of the issues around what makes a particular set of features 'good', from a deep learning perspective.