Creating shared vertex normals on GPU I've mostly succeeded at porting an implementation of Marching Cubes from CPU over to OpenGL compute shaders, but I haven't tackled normals yet and wondering the best way to go about it.

My implementation deals specifically with binary valued fields (I'm trying to model 3D fractal functions that don't have a distance estimator yet), so gradient and forward differences methods won't work. I have shared vertices working, and my CPU implementation uses Quilez's method described here to accumulate face normals onto each neighbouring vertex.

I could just port this implementation over to another shader, but the problem I see with this is the massive number of atomics needed. Since we can only use atomics on scalar integer types, and I cant think of a way to pack 3 signed ints into 1 in a summable way, that means 3 axes * 3 vertices = 9 atomic adds per shader invocation. They'll be spread throughout memory of course so it's not like hitting a single atomic counter 9 times, but it still seems like a hell of a lot.

The other alternative is to run a shader invocation per-polygon and build the face normal list (i could probably pack to x10y10z10 this way), then a shader per-vertex to accumulate all the normals of neighbouring faces. This would be an enormous memory hog though, the storage space of the face indices would need 12 int per vertex to deal with the worst case. There's also the problem of how to write into this storage without again resorting to atomics to work out how many faces have already been written to a particular vertex.

Anyone have any better ideas on how to do this?
For an nVidia only solution you can use floating point atomic add intrinsics (like NvInterlockedAddFp32) Unlocking GPU Intrinsics in HLSL | NVIDIA Developer

I tried this on 80.000 vertex mesh and it's quite fast (something like 1 or 2 ms on a GTX980M, if I remember correctly)

Just beware of compiling your shaders in release for the intrinsics to work (due to nvidia bug/limitation)

Also beware of vertex splits (due to UV discontinuities for example), you'll have to handle them or else you'll have unwanted hard edges at UV seams.