Slow datetime parsing in Pandas
and 2. In my experience, if processing time is not critical for your study (say you process the data once and then you run your analysis) then I would recommend you parse the dates using pd.to_datetime() and others after you have read in the data.
anything that will help Pandas reduce the set of possibilities about the types in your data WILL speed up the processing. That make sense. The more precise you are, the faster will be the processing.
These questions about datetime parsing in pandas.read_csv() are all related.
Question 1
The parameter infer_datetime_format is False by default. Is it safe to set it to True? In other words, how accurately can Pandas infer date formats? Any insight into its algorithm would be helpful.
Question 2
Loading a CSV file with over 450,000 rows took over 10 mins when I ran pd.read_csv("file.csv", parse_dates = ["Start", "End"])
However it took only 20 seconds when I added the parameters dayfirst = True and infer_datetime_format = True. Yet if either was False, it took over 10 mins.
Why must both be True in order to speed up datetime parsing? If one is False but not the other, shouldn't it take strictly between 20 sec and 10 mins? (The answer to this question may well be the algorithm, as in Question 1.)
Question 3
Since dayfirst = True, infer_datetime_format = True speeds up datetime parsing, why is it not the default setting? Is it because Pandas cannot accurately infer date formats?