Categorical and non-categorical data in the same column
If you provide more information about the detail of why there is no data for some data points that would make it easier. 

That being said, I would split it into three columns as follows:

col_1: it includes 0 and 1 >>> 1 for those samples that have a value of -4 in the original column. For the rest of the samples: 0 

col_2: it includes 0 and 1 >>> 1 for those samples that have a value of -1 in the original column. For the rest of the samples: 0 

col_3: the real values of the original columns. If the value is -4 or -1, there are two options as follows:
1. Use 0 for those samples
2. If they are missing values replace them using one of the methodologies used for replacing the missing values.