How to extract a smooth contour from a set of points in 3D?
From your description it sounds like surface reconstruction methods such as Poisson Reconstruction would fit your description. These algorithms already have the necessary smoother built-in, mainly in the form of a regularizer. To have slightly more accuracy, in general I prefer the following algorithm:


  SSD: Smooth Signed Distance Surface Reconstruction, F. Calakli and G. Taubin, Computer Graphics Forum, Vol. 30, No. 7, 2011, http://mesh.brown.edu/ssd/


There is a nice Adaptive Solvers repository containing many popular methods under a single roof:
https://github.com/mkazhdan/PoissonRecon

I strongly recommend to take a look at this. Here is how you would like to use it:


Export your point set and the surface normals to a 3D file : sample.ply
Run (make sure to play with the depth value):

SSDRecon --in sample.ply --out sample_reconstruction.ply --depth 10 --density


This will generate a reconstruction but if you have an open surface the reconstruction will contain undesired extrapollations. So move to (3). If the surface is perfectly closed, then the output will already be good in that stage.


If you have an open surface then run:

SurfaceTrimmer --in sample_reconstruction.ply --out sample_reconstruction_trimmed.ply --trim 7


Again, make sure to play with the trimming threshold. This will give you very good reconstructions such as (taken from here): 



Note that for all these algorithms to work, surface normals are essential. Luckily, they can be computed from the data using local plane fits. Moreover, one also needs to ensure the correct signs of the normal directions to properly orient the surface. In the case that a single camera captures the data, the canonical choice is to flip all the surface normals so that they make an acute angle with the viewing direction.