Converting a nominal attributes to numerical ones in data set
By converting a nominal attribute to a single numeric attribute as you described, you are implicitly introducing an ordering over the nominal labels which is a bad representation of the data, and can lead to unwanted effects from a classifier. Does it make sense to say that UDP should be inbetween TCP and ICMP? (no!) Imagine you are training a $k$-NN model on this data. It doesn't make sense to say that ICMP should be "further away" from TCP than UDP, but if you adopted the mapping that you suggested, the representation of the data has this assumption built-in. Alternatively, what if you are training a decision tree-based model? Usually, in decision trees, binary split points are chosen for numeric attributes. There could be some randomness in your training data where splits at certain values of the numeric attribute results in overfitting to noise.

Typically when converting a nominal attribute to numeric, one numeric attribute per nominal label is created. Each attribute is set to one if the corresponding nominal label is set, and zero otherwise. For example, if a nominal attribute called protocol has labels {tcp, udp, icmp}, then this dataset:

$$
\begin{array}{ccl}
\text{inst.} &amp; \text{protocol} &amp; \text{other attributes} \\ \hline
1 &amp; \text{tcp} &amp; \dots \\
2 &amp; \text{icmp}&amp; \dots \\
3 &amp; \text{icmp}&amp; \dots  \\
\vdots &amp; \vdots &amp; \ddots
\end{array}
$$

could be converted as follows:

$$
\begin{array}{ccccl}
\text{inst.} &amp; \text{tcp} &amp; \text{udp} &amp; \text{icmp} &amp; \text{other attributes} \\ \hline
1 &amp; 1 &amp; 0 &amp; 0 &amp; \dots \\
2 &amp; 0 &amp; 0 &amp; 1 &amp; \dots \\
3 &amp; 0 &amp; 0 &amp; 1 &amp; \dots \\
\vdots &amp; \vdots &amp; \vdots &amp; \vdots &amp; \ddots \\
\end{array}
$$

This is what the NominalToBinary filter does in WEKA. As you mention, the downside of this is that a large number of additional attributes can be introduced if the number of distinct nominal values is high. 

If the dimensionality is too high after the conversion, you may want to consider using a dimensionality reduction technique such as random projection, PCA, t-SNE, etc. Note that this will reduce the interpretability of your model. You could also use feature selection techniques to remove some of the less useful attributes. It is possible that some of the nominal labels are not useful for your model, and you will improve performance by removing them. Another thing you could try is to use your domain knowledge to reduce the number of categories. For example, TCP and UDP are both transport protocols, maybe for your application the distinction between TCP and UDP is not that important and you can put instances with protocol $\in$ {tcp, udp} into a new category, removing the old ones.