title: Overshadow data in aws glue studio
body: <p>I am constructing an ETL process in AWS Glue Studio where I get the data in a bucket s3 to remove some fields. After this process, I need to use a Custom Transformation to overshadow some data and then save it in a new s3 bucket.</p>
  <p>When I use a Custom Transformation a block code is shown. So I need to create my code there.</p>
  <p>My question is about it, how can I use it to construct my transformation, more specifically about input and output data, because I don't understand this process on AWS.</p>
  <p>What I did at the moment was create the function that will manipulate this data. I get my fields using dfc['my_field'] and applied the transformation, but I don't understand what I need to return and how can I save it in an s3 bucket. below you can see my function.</p>
  <pre><code>def MyTransform (glueContext, dfc) -&gt; DynamicFrameCollection:
  import random
  import hashlib
  from pyspark.sql.types import IntegerType
  from pyspark.sql.functions import col, lit
  
  df = dfc.select(list(dfc.keys())[0]).toDF()
  print('printing key0')
  df.show(3)
  print('printed key0')
  print('imprimindo o tipo de df')
  print(type(df))
  
  def encrypt_phone(df):
      '''
       Criptografa dados de telefone
      '''
      if df.withColumn('phone_main', df('phone_main')) is not None:
          hl = hashlib.sha256()
          hl.update(df.withColumn('phone_main', df('phone_main'))).encode().hexdigest()
          hl.update(df.withColumn('phone_alternative', df('phone_alternative'))).encode().hexdigest()
      return df
  
  # Criar um UDF para encriptar os dados
  from pyspark.sql.functions import udf
  encrypted_udf = udf(encrypt_phone)
  print(encrypted_udf)
  
  # Renomear coluna phone_main e phone_alternative
  df.withColumnRenamed('phone_main', 'old_phone_main') \
                      .withColumnRenamed('phone_alternative', 'old_phone_alternative')
  
  # Criar novas colunas para dados transformados
  new_df = encrypt_phone(df)
  encrypted = df.withColumn('phone_main', new_df('phone_main'))
  encrypted = df.withColumn('phone_alternative',new_df('phone_alternative'))
  print(encrypted)
  
  # Remover campos sensíveis
  encripted_df = df.drop('cpf_cnpj', 'name', 'nickname', 'old_phone_main', 'old_phone_alternative')
  print(df_encrypted)
  
  # Mascarar email
  print(encripted_df.withColumn('email', df('email')))
  if encripted_df.withColumn('email', df('email')) is not None:
      masked_df = encripted_df.withColumn('email', lit('****@*****'))
      
  # Converter spark dataframe para Glue Dynamic Frame e retornar como um dataframe collection
  df_overshadow = DynamicFrame.fromDF(masked_df, glueContext, &quot;df_overshadow&quot;)
  
  return (DynamicFrameCollection({&quot;CustomTransform0&quot;: df_overshadow}, glueContext))
  </code></pre>
  <p>My process on glue studio to be structured like this image:</p>
  <p><a href="https://i.stack.imgur.com/ybqt3.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/ybqt3.png" alt="enter image description here" /></a></p>
  <p>But, when I try to save an error is shown:</p>
  <pre><code>Parent node overshadow outputs a collection, but node Process-Client_PFS does not accept a collection.
  </code></pre>
  <p>I need to apply the transformation and save the data. How can I do it?</p>