When does cache get expired for a RDD in pyspark?
It will not expire until Spark is out of memory, at which point it will remove RDDs from cache which are used least often. When you ask for something that has been uncached it will recalculate the pipeline and put it in cache again. If this would be too expensive, unpersist other RDDs, don't cache them in the first place or persist them on your file system.