Subdivision-surface giving strange geometry with tris
Tris and subsurf don't blend well. I would suggest reworking that mesh, try using quads wherever possible. For a hard surface object, tris are ok to a point but once you start to subdivide and smooth it, it quickly gets out of hand. The 'strange geometry' is more than likely caused by the tris pinching together at one point. If you go into edit mode, use wireframes and look at what the modifier is doing. It is mutilating your mesh trying to subdivide those triangular faces.
An alternative
The way I normally go about this and one way to do it is to add the circle and a plane to the scene, delete an edge of the plane so the face goes away and then add it back. You should now have something like this below. (This is the wireframe)

This next step will be unique to every case as it depends how many vertices the circle has. You generally want to estimate at this point and select an even or so amount of vertices and subdivide one side of the plane and join these together. You can manually do this for the entire plane side by side or automate it with the mirror modifier or duplicating and rotating and removing doubles etc.. depends what works best for you.
At this point, it will be easy to see if you are off as the mesh will look odd. You generally want to aim for something that gets nice even spaces and looks symmetrical to the other sides to a point.

You now have a mesh with a circle in it with nice clean faces so you can easily add more geometry as needed and subdivide as much as you want.


For more tips see: How to fill a hole in face?
Edit:
As Aldrik mentions in the comments below, another option is to subdivide the surface to get some geometry and use the To Sphere function with Shift + Alt + S. This isn't always perfect in my experience but can work at times.