Why spark regexp_extract too slow when we compare with python re
Probably more context is needed to give an specific answer, but what I can infer from what you said is the following:
I would think that it depends on the size of the data and how are the partitions in spark. As spark is parallelizing, probably in not huge amounts of data, regular python functions will work faster, but not in huge amounts of data were parallelization is more handy.
Recently i came across a requirement where i tried to change python re with pyspark regexp_extract, the reason behind change re to pyspark regexp_extract is spark is more faster. by comparing the process speed with pyspark  and re process I concluded like re is faster than pyspark regexp_extract. Is there any specific reason that cause pyspark regexp_extract is slow.
Thanks in advance