What graphics techniques can be used to simulate a color nebula effect?
I have no knowledge of SceneKit whatsoever, but the effect appears to be a simple post-process Gaussian blur of the app icon grid, with the blur radius increasing over time (i.e. increasing the blur amount) during the transition, with a touch of colour correction tacked on (which causes the darkening). It should be easy enough to implement in any graphics API that allows rendering to textures and performing image filtering on them (for instance, in OpenGL you could use framebuffer objects to achieve this).

You can have a look at an example WebGL implementation here. It's worth noting that it is a separable filter; meaning, it's a rectangle-shaped filter, but you can avoid sampling the same pixels (overlapping neighbouring samples) multiple times by splitting it into two passes – a horizontal and a vertical one – to the same effect.