How does log in spark stage/tasks help in understanding actual spark transformation it corresponds to
Break your execution down. It's the easiest way to understand where the error might be coming from. Running a 500+ line of code for the first time is never a good idea. You want to have the intermediate results while you are working with it. Another way is to use an IDE and walk through the code. This can help you understand where the error originated from. I prefer PyCharm (Community Edition is free), but VS Code might be a good alternative too.
Often during debugging Spark Jobs on failure we can find the appropriate Stage and task responsible for the failure such as String Index Out of Bounds exception but it becomes difficult to understand which transformation is responsible for this failure.The UI shows information such as Exchange/HashAggregate/Aggregate but finding the actual transformation responsible for this failure becomes really difficult in 500+ lines of code, so how should it be possible to debug Spark task failures and tracing the transformation responsible for the same?