Q: Wrong combination of transparency group soft mask and PNG image mask I have a TransparencyGroup-based soft mask as described in 7.5.4 of the PDF specification. It works all right when the image I apply the mask to is a JPG but fails when it has its own mask like a PNG.
doc = new Document(new Rectangle(ToPdf(210), ToPdf(297)));
pdf = PdfWriter.GetInstance(doc, new FileStream("test.pdf", FileMode.Create));
pdf.PdfVersion = PdfWriter.VERSION_1_4;
pdf.CompressionLevel = PdfStream.NO_COMPRESSION;

doc.Open();
var dc = pdf.DirectContent;

dc.Rectangle(0, 0, ToPdf(210), ToPdf(297));
dc.SetColorFill(BaseColor.BLUE);
dc.Fill();

dc.SaveState();
var mask = Image.GetInstance("mask.jpg");
mask.ScaleAbsoluteHeight(ToPdf(100));
mask.ScaleAbsoluteWidth(ToPdf(100));
mask.SetAbsolutePosition(0, 0);

var transparency = dc.CreateTemplate(ToPdf(100), ToPdf(100));
transparency.Group = new PdfTransparencyGroupEx { ColorSpace = PdfName.DEVICEGRAY };
transparency.AddImage(mask);

var softmask = new PdfSoftMask(PdfName.MASK) {
  Subtype = new PdfName("Luminosity"),
  Group = transparency.IndirectReference,
};
dc.SetGState(new PdfGStateEx {
  SoftMask = softmask,
  AlphaIsShape = false,
});

var picture = Image.GetInstance("test.png"); // or test.jpg
//picture.Smask = false;
picture.ScaleAbsoluteHeight(ToPdf(100));
picture.ScaleAbsoluteWidth(ToPdf(100));
picture.SetAbsolutePosition(0, 0);
dc.AddImage(picture);
dc.RestoreState();

doc.Close();

The mask is simply a grayscale image with a fountain fill:

With the PNG, the image will appear with an erroneous background:

Helpers:
public static float ToPdf(double mm) => (float)(mm / 25.4 * 72.0);

public class PdfGStateEx : PdfGState {
  public PdfObject SoftMask {
    set => Put(PdfName.SMASK, value);
  }
}

public class PdfTransparencyGroupEx : PdfTransparencyGroup {
  public PdfName ColorSpace {
    set => Put(PdfName.CS, value);
  }
}

public class PdfSoftMask : PdfDictionary {
  public PdfSoftMask(PdfName type)
    : base(type) {
  }

  public PdfName Subtype {
    set => Put(PdfName.S, value);
  }

  public PdfIndirectReference Group {
    set => Put(new PdfName("G"), value);
  }

  public PdfArray BackdropColor {
    set => Put(PdfName.BC, value);
  }
}

Test files


*

*PDF with a JPG

*PDF with a PNG, SMask

*PDF with a PNG, Mask
Some analysis
in all cases, the image is embedded correctly:
/GS1 gs
q 283.46 0 0 283.46 0 0 cm /img1 Do Q

where the soft mask dictionary is:
<<
  /AIS false
  /SMask
  <<
    /G 1 0 R
    /S /Luminosity
    /Type /Mask
  >>
>>

referencing the transparency group XObject:
<<
  /BBox [0 0 283.46 283.46]
  /FormType 1
  /Group
  <<
    /CS /DeviceGray
    /S /Transparency
  >>
  /Length 38
  /Matrix [1 0 0 1 0 0]
  /Resources
  <<
    /XObject
    <<
      /img0 2 0 R
    >>
  >>
  /Subtype /Form
  /Type /XObject
>>

No difference so far. One of the PNG versions actually clear the SMask entry, then the inherent mask of the PNG disappears, this is to be expected.
So, this will be likely the case because the soft mask of the image will override the soft mask in the graphics state. Now the question boils down to: is there any support in iText to blend the two masks (one from the PNG, one from my own) or do I need to do this separately prior to feeding it to iText?

A: 
So, this will be likely the case because the soft mask of the image will override the soft mask in the graphics state.

Indeed, soft masks set via the graphics state and soft masks set with an image essentially operate the same way, and at the same time there can be only one active soft mask.

Now the question boils down to: is there any support in iText to blend the two masks (one from the PNG, one from my own) or do I need to do this separately prior to feeding it to iText?

No, there is no support in iText to blend the two masks; iText only reads images from external sources and embeds them into a PDF or (during extraction) reads them from the PDF for export to some external target, it does not do further processing.
But still you don't have to do this separately prior to feeding it to iText, you can leave this blending to the PDF viewer! While at the same time there can be only one active soft mask, you can make use of nested transparency groups to combine different transparency effects.
In your code simply replace
dc.AddImage(picture);

by
var group = dc.CreateTemplate(ToPdf(100), ToPdf(100));
group.Group = new PdfTransparencyGroupEx();
group.AddImage(picture);
dc.AddTemplate(group, 0, 0);

(i.e. instead of adding the image with its soft mask to the content directly, add it to a separate transparency group and add that transparency group to the content).
The result will change like this:

-->


By the way, you can improve the quality of the transparency mask (in particular at higher resolutions or zoom levels) by using an actual continuous gradient instead of a bitmap with a gradient. Simply replace 
transparency.AddImage(mask);

by
PdfShading shading = PdfShading.SimpleAxial(pdf, 0, 0, ToPdf(100), 0, BaseColor.BLACK, BaseColor.WHITE);
transparency.PaintShading(shading);

