List of word to list of index in a vocabulary
When using NumPy, stick to NumPy functions where possible instead of going via pure Python and back again (which is usually slower). So instead of getting the unique elements of an array by writing np.array(list(set(...))), call numpy.unique.
The expression np.argwhere(vocs[j]==x[i,j]) has to search the whole vocabulary for the word x[i,j], and this has to be done for every feature in every sample. This means that if there are \$s\$ samples and \$f\$ features, and each feature has \$w = O(s)\$ words, the overall runtime is proportional to \$sfw = O(s^2f)\$. That is, it's quadratic in the number of samples. This is why it takes so long.

To avoid this, we need to construct, for each feature, an inverse mapping from words in the samples to their indexes in the vocabulary. How do we construct such inverse mappings? Well, looking at the numpy.unique documentation, we find that it takes a keyword argument:


  return_inverse : bool, optional
  
  If True, also return the indices of the unique array (for the specified axis, if provided) that can be used to reconstruct ar.


These inverse arrays are exactly what you need, and so your code becomes:

vocs, inverses = zip(*(np.unique(feature, return_inverse=True) for feature in x.T))
x_new = np.vstack(inverses).T