Reliable way to verify Pyspark data frame column type If I read data from a CSV, all the columns will be of "String" type by default. Generally, I inspect the data using the following functions which gives an overview of the data and its types

df.dtypes
df.show()
df.printSchema()
df.distinct().count()
df.describe().show()


But, if there is a column that I believe is of a particular type e.g. Double, I cannot be sure if all the values are double if I don't have business knowledge and because


I cannot see all the values (millions of unique values) 
If I explicitly cast it to double type, spark quietly converts the type without throwing any exception and the values which are not double are converted to "null" - for example


Code:

from pyspark.sql.types import DoubleType

changedTypedf = df_original.withColumn('label', df_control_trip['id'].cast(DoubleType()))


What could be the best way to confirm the type of column then?
If you don't have business knowledge, there is no way you can tell the correct type, and no way you can 'confirm' it. You can at most make assumptions about your dataset and your dataset only, and you for sure have to inspect every value.

In your example, you created a new column label that is a conversion of column id to double. You could count all rows that are null in label but not null in id. If this count is zero you can assume that for this dataset you can work with id as a double. That doesn't necessarily mean that in a new dataset the same will be true for column id.