parquet format: advise on log content I'm using a python script to log IO of a grid job.
the log is formatted like this: 

timestamp;fullpath;event;size
1526994189.49;/tmp/folder/;IN_ISDIR;6
1526994189.49;/tmp/folder2/File;IN_ACCESS;36


Those files are millions of line long. I'm using Spark to generate graphs and detect anomaly in job IO.
But before doing that I need to insert the job ID and the jobname to the column making the file :  

timestamp;fullpath;event;size;jobid;jobname
1526994189.49;/tmp/folder/;IN_ISDIR;6;123456;afakejobname
1526994189.49;/tmp/folder2/File;IN_ACCESS;36;123456;afakejobname


The thing is I'm new to big Data technologies and I would like to know if using parquet format is it better to put both jobname and jobid
or knowing that I have only 15 different jobname and jobid in the same log is it better to convert it on the fly using SparkSQL and make a join to a very small table with just jobname;jobid and put only the jobid in my log.
Your second solution sounds great, and to use it in the most effective way, look for broadcast variables in Spark. 

Then, you're using the features available in spark to optimize it.

Reference: https://jaceklaskowski.gitbooks.io/mastering-apache-spark/content/spark-broadcast.html