I am running a PySpark script on AWS EC2. It runs very well on Jupyter notebook, however when I run it on an IPython shell, it gives import error. It looks so weird! Can anybody help, please. 
Here's a snippet of the code:

```
from  __future__ import division
 from pyspark import SparkContext
 from pyspark.sql import SQLContext,SparkSession
 from pyspark.sql.functions import lower, col,trim,udf,struct,isnan,when
 from pyspark.sql.types import StructType, StructField, IntegerType, 
 StringType,FloatType,ArrayType,Row
 from pyspark.sql.functions import lit
 import gc
 import time
 import pandas as pd
 from collections import defaultdict
 import numpy as np

 sc = SparkContext(appName="Connect Spark with Redshift")
 sql_context = SQLContext(sc)
 sc._jsc.hadoopConfiguration().set("fs.s3n.awsAccessKeyId", 'xyz')
 sc._jsc.hadoopConfiguration().set("fs.s3n.awsSecretAccessKey", 'pqr')

 spark=SparkSession.builder.master("local").appName("Users").getOrCreate()
 users=pd.read_pickle(candidate_users_path)
 sqlCtx = SQLContext(sc)
 users = sqlCtx.createDataFrame(users)
 users.count()

```

It gives error at the import statement (2nd line). Funny part is it's running so beautifully on Jupyter notebook launched from same location. And, the same import statement is working if I just execute that import statement in IPython. In my understanding, this EC2 acts as worker and master, then how can it be not available in the worker?
`Py4JJavaError: An error occurred while calling o57.count.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 5 in stage 0.0 failed 1 times, most recent failure: Lost task 5.0 in stage 0.0 (TID 5, localhost, executor driver): org.apache.spark.SparkException:
Error from python worker
 ImportError: cannot import name 'SparkContext'
PYTHONPATH was:
 /home/ubuntu/spark-2.4.3-bin-hadoop2.7/python/lib/pyspark.zip:/home/ubuntu/spark-2.4.3-bin-hadoop2.7/python/lib/py4j-0.10.7-src.zip:/home/ubuntu/spark-2.4.3-bin-hadoop2.7/jars/spark-core_2.11-2.4.3.jar
org.apache.spark.SparkException: No port number in pyspark.daemon's stdout
 at org.apache.spark.api.python.PythonWorkerFactory.startDaemon(PythonWorkerFactory.scala:204)
 at org.apache.spark.api.python.PythonWorkerFactory.createThroughDaemon(PythonWorkerFactory.scala:122)
 at org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:95)
 at org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:117)`