QST:
I am using the below code to merger the multiple CSV files in the folder. The code is working fine for smaller dataset but for bigger data.

As the total number of rows is 51163632 but as a result, in big_frame I am getting only 20883611 rows.

Is there a limit of pandas dataframe to store the values?

big_frame = pd.DataFrame()

filename= []
num_comments=0
for file in os.listdir():
    print(file)
    if file.endswith('.CSV'):
        filename.append(file)
        num_lines = sum(1 for line in open(file, encoding="utf8"))
        #print(num_lines)
        num_comments = num_comments + num_lines
        print(num_comments)
        try:
        #df = pd.read_csv(file, error_bad_lines=False, sep=',', header=None, encoding='utf-8')
            df = pd.read_csv(file, error_bad_lines=False, delimiter=',', header=None, encoding='utf-8')
        except:
            continue
        big_frame = big_frame.append(df, ignore_index=True)