Unformatted data entries
Those symbols you are talking about are hex values. They could actually be useful to keep in the dataset, as the tcpdump tool outputs them if the right options are set.

Despite that, you can replace occurrences of them using pandas and a simple regular expression. A regular expression that matches hex values is: 0[xX][0-9a-fA-F]+ - see a live demo of that here.

I downloaded one of the CSV files and did it like this:

import pandas as pd

filename = "./UNSW-NB15_1.csv"
df = pd.read_csv(filename, nrows=5000)     # read first 5000 lines
df.to_html("with_hex.html")                # open in a browser to inspect (e.g. line 211)


Now use the DataFrame replace and set the regex flag, and insert whatever you want to replace the matches with (here "NaN").

clean = df.replace("0[xX][0-9a-fA-F]+", value="NaN", regex=True)


I used this to have a look at the cleaned versions of the data quickly in a browser (notice line 211):

clean.to_html("clean.html")




You can use the same trick to replace almost anything, so your fields with '-' can be replaced with:

# "inplace" means it overwrites the values, no new dataframe is returned
clean.replace("-", "NaN", regex=False, inplace=True)