# PySpark Refer Column Name With Dot (.)

• Post author:

Problem: I have a PySpark (Spark with Python) DataFrame with a dot in the Column names, could you please explain how to access/refer to this column with the dot by using `<a href="https://sparkbyexamples.com/pyspark/pyspark-withcolumn/">withColumn()</a>` or `<a href="https://sparkbyexamples.com/pyspark/select-columns-from-pyspark-dataframe/">select()</a>` transformations. Below is the PySpark DataFrame with column `name.fname` with dot.

``````
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName('SparkByExamples.com').getOrCreate()
data=[("James",23),("Ann",40)]
df=spark.createDataFrame(data).toDF("name.fname","gender")
``````

## PySpark DataFrame Column Name with Dot (.)

Solution: Generally as a best practice column names should not contain special characters except underscore (_) however, sometimes we may need to handle it.

In order to access PySpark/Spark DataFrame Column Name with a dot from `wihtColumn()` & `select()`, you just need to enclose the column name with `backticks (`)`

Using Column Name with Dot on `select()`.

``````
from pyspark.sql.functions import col
df.select(col("`name.fname`")).show()
df.select(df["`name.fname`"]).show()
+----------+
|name.fname|
+----------+
|     James|
|       Ann|
+----------+
``````

Using Column Name with Dot on `withColumn()`

``````
df.withColumn("new_col",col("`name.fname`").substr(1,2)).show()
+----------+------+-------+
|name.fname|gender|new_col|
+----------+------+-------+
|     James|    23|     Ja|
|       Ann|    40|     An|
+----------+------+-------+
``````

Using on `filter()`, for more example refer to PySpark Filter() Usage & Examples

``````
df.filter(col("`name.fname`").startswith("J")).show()
+----------+------+
|name.fname|gender|
+----------+------+
|     James|    23|
+----------+------+
``````

## Replace All Column Names from Dot (.) to underscore (_)

Have a column name with a dot leads us into confusion as in PySpark/Spark dot notation is used to refer to the nested column of the struct type. so if possible try to replace all column names with dot to underscore before processing it.

``````
new_cols=(column.replace('.', '_') for column in df.columns)
df2 = df.toDF(*new_cols)
df2.show()

+----------+------+
|name_fname|gender|
+----------+------+
|     James|    23|
|       Ann|    40|
+----------+------+
``````

Happy Learning !!

### NNK

SparkByExamples.com is a Big Data and Spark examples community page, all examples are simple and easy to understand and well tested in our development environment Read more ..
{"ft_lang_label":"__label__en","ft_lang_prob":0.5788289,"math_prob":0.59021723,"size":2201,"snap":"2022-27-2022-33","text_gpt3_token_len":545,"char_repetition_ratio":0.21165225,"word_repetition_ratio":0.016129032,"special_character_ratio":0.34348023,"punctuation_ratio":0.16304348,"nsfw_num_words":0,"has_unicode_error":false,"math_prob_llama3":0.9517657,"pos_list":[0],"im_url_duplicate_count":[null],"WARC_HEADER":"{\"WARC-Type\":\"response\",\"WARC-Date\":\"2022-08-08T22:52:29Z\",\"WARC-Record-ID\":\"<urn:uuid:8701c6a8-827b-4747-8208-7ef1b7db4624>\",\"Content-Length\":\"290312\",\"Content-Type\":\"application/http; msgtype=response\",\"WARC-Warcinfo-ID\":\"<urn:uuid:0a0c04db-bc30-431e-a069-1806775a1cdc>\",\"WARC-Concurrent-To\":\"<urn:uuid:e07194c4-8c1a-4a6b-b985-007db96a6a9f>\",\"WARC-IP-Address\":\"172.67.176.248\",\"WARC-Target-URI\":\"https://sparkbyexamples.com/pyspark/pyspark-refer-column-name-with-dot/\",\"WARC-Payload-Digest\":\"sha1:ZRVQYRDXSHRTWFN4BZWXA7YKOEGIVRPM\",\"WARC-Block-Digest\":\"sha1:Y3NGHZUQFRFWP5U772QAOJO4LL4BLWOY\",\"WARC-Identified-Payload-Type\":\"text/html\",\"warc_filename\":\"/cc_download/warc_2022/CC-MAIN-2022-33/CC-MAIN-2022-33_segments_1659882570879.1_warc_CC-MAIN-20220808213349-20220809003349-00658.warc.gz\"}"}