What type of photosynthesis is performed by phytoplankton?
I'm not sure that all phytoplankton perform the same type of photosynthesis. 

Originally, people thought that they all performed C4, on the basis of genome sequencing, which revealed the presence of genes important for C4 photosynthesis.  However, experiments on individual species seemed to indicate that the phytoplankton were performing multiple types of photosynthesis (source).

EDIT: CAM is unlikely, as that is an adaptation for dry environments.

EDIT 2: and of course, some phytoplankton appear to have both C3 and C4 pathways. In short, you can't generalize.