Spark DataFrame "Limit" function takes too much time to display result
The reason is the way limit and the show is implemented under the hood. Show just reads the first 20 (first n) rows, which limit reads the whole data before showing it. Refer this answer on StackOverflow - link