multivariate clustering, dimensionality reduction and data scalling for regression I have a dataset with approximately 20000 observations consisting of 40 independent and 1 dependent variable. My initial objective is to develop a model that will predict the dependent variable. I have tried several models and applied linear regression and other algorithms such as Random Forests, of course by splitting the dataset into training and testing sets.

Unfortunatelly I cannot get any meaningful results; I have very large errors. I believe there is something "messy" with the dataset, so I have decided to do some clustering first and then apply regression within each cluster. Considering that my dependent variable may exhibit a lot of variation I believe I should do clustering with all variables (dependent and independent), so as each cluster will have similar values of my dependent variable. I have tried to apply Kmeans and I faced several problems. First of all, it seems I cannot identify the right number of clusters. The "elbow" method gives an unclear number and when I use it with less data (about 2000 observations) I get something like this:



I also had similar problems with hierarchical clustering. I have already tried to apply regressions within the clusters identified, but the results are still very poor.

Right now I believe I should possibly use some kind of "weight" to my data, in order to put more weight on the dependent variable when I do clustering, since I believe that this is the problem. Hence, my questions here are:


is there a way/algorithm where I adjust weights in the variables to be clustered?


Moreover I am confused with two more issues:


data scaling:
is it necessary to scale the data before clustering? does this guarantee more accurate results? when do we scale the data? 
dimensionality reduction:
I have read a lot about principal component analysis and dimensionality reduction, but I am still confused. Again; is this necessary? how many variables are too many to consider applying PCA? are 10 variables too many? or maybe 20? or 50? when should we apply dimensionality reduction? a problem with PCA is that I would still need my original variables to extract the coefficients after regression, while to my understanding with PCA I cannot do that.


This question is more about discussion in order to understand some particular concepts and find a solution to my problem and does not refer to coding issues. Any example and/or references would be appreciated though. I am coding in R.
Great question, I will try to answer the aspects related to dimensionality reduction mentioned above.
$Dimensionality\: Reduction:$ The number of dimensions which you want to keep after doing PCA is an experimental value you can experiment with the number of dimensions and check you results. Although you have mentioned all 40 features are independent i would still ask you to do a correlation analysis of the variables.

Pca removes these correlated features and gives you a set of features which amount to explain most of your data. One advantage of dimensionality reduction is in regression. Correlated features often are a cause of multicollinearity. Doing a dimensionality reduction helps us get rid of this problem. Also once we have a reduced set of features we can apply the cluster analysis. The reason is K-means calculates the l2- distance between data points. in very high dimensions the concept of euclidean distance becomes less useful because of the curse of dimensionality. (probably the reason of the problems with your elbow curve.)

Hence bringing down the number of features using pca and clustering later will give a better idea of the groupings of the data.
The problem which you mention about PCA can be solved in fact it is not an issue at all. You can do regression with the features obtained by the data and only take the coefficients of the reduced feature set.Some other techniques which you might want to look at are regularization in regression.
For example L1-regularization helps in feature selection and helps with correlated variables.