I have.ASC data files in a directory being written into it by another program and the data look something like this
[![enter image description here](https://i.stack.imgur.com/NEUqI.png)](https://i.stack.imgur.com/NEUqI.png)

the data is in multiple files where comments start with '/', the header appears multiple times in the file but I want to keep only one header for all the data rows.

the ultimate goal is to load the whole data file as soon as it's written into the directory into a Pandas Dataframe. I have tried simple Pandas read\_csv as only on the one file

```
import pandas as pd
df=pd.read_csv("demo.txt", header = None, sep = "\s+",comment='/')
df.head()

```

and got the following result:

[![enter image description here](https://i.stack.imgur.com/BJNCW.png)](https://i.stack.imgur.com/BJNCW.png)

later I tried to use the traditional python read file operation which works but partially as you can see that it's skipping many rows.

```
f = open("demo.txt", "r")
for i in f:
    if not (f.readline(1)=='/'):
        f2 = open("demofile2.txt", "a")
        f2.write(f.readline())

f2.close()

```

[![enter image description here](https://i.stack.imgur.com/HC1zP.png)](https://i.stack.imgur.com/HC1zP.png)

```
the algorithm could be:
read one file or multiple files or as soon as a new file written into the directory
read it directly as .ASC if not change to . TXT a
keep the headers in the first row and discard all the comments.

```

Note: i have manually changed the type .ASC to.TXT,

Update: trying to add a smaller dataset which can be locally copy-pasted

```
/comments start

/comments end
/id h1  h2  h3  Date        h5
0   1   41  0   12/4/2018   0
1   1   0   0   12/4/2018   0
2   1   0   0   12/4/2018   0
3   1   0   0   12/4/2018   0
4   1   90  0   12/4/2018   0
/comments start

/comments end                   
/id h1  h2  h3  Date        h5
5   1   41  0   12/4/2018   0
6   1   0   0   12/4/2018   0
7   1   0   0   12/4/2018   0
8   1   0   0   12/4/2018   0
9   1   90  0   12/4/2018   0

```

want it to look like this:

```
id  h1  h2  h3  Date        h5
0   1   41  0   12/4/2018   0
1   1   0   0   12/4/2018   0
2   1   0   0   12/4/2018   0
3   1   0   0   12/4/2018   0
4   1   90  0   12/4/2018   0
5   1   41  0   12/4/2018   0
6   1   0   0   12/4/2018   0
7   1   0   0   12/4/2018   0
8   1   0   0   12/4/2018   0
9   1   90  0   12/4/2018   0

```

Notice that this pattern repeats in the file several times, i.e comments,->headers->data
comments->headers->data and so on. and there are multiple files in the directory.