Dependency Overkill in setup.py
The extensions dependencies [defined](https://github.com/pandas-dev/pandas/blob/master/setup.py#L471) in `ext_data` are passed to the `distutils.extension.Extension` constructor as

```
extensions = []

for name, data in ext_data.items():
    sources = [srcpath(data['pyxfile'], suffix=suffix, subdir='')]
    pxds = [pxd(x) for x in data.get('pxdfiles', [])]
    if suffix == '.pyx' and pxds:
        sources.extend(pxds)

    sources.extend(data.get('sources', []))

    include = data.get('include', common_include)

    obj = Extension('pandas.%s' % name,
                    sources=sources,
                    depends=data.get('depends', []),
                    include_dirs=include,
                    extra_compile_args=extra_compile_args)

    extensions.append(obj)
```

_None of the `data` dicts has a `include` key_, so they all end up passing `include=common_include=['pandas/_libs/src/klib', 'pandas/_libs/src']`. As a result(?) a bunch of the dependencies that _are_ listed can be removed without breaking the build.  e.g. I can completely get rid of `lib_depends` and every appearance of `_libs/src/util' without any apparent consequences.

I'm just speculating because I don't really get how depends vs include_dirs works, but it seems like having `common_include` added to everything is massive overkill.  For most of these exts they only need a couple of the files from _libs/src/.  But poking at it, I have not figured out what combination of args is needed to specify e.g. `reshape.pyx cimports src/util.pxd which needs src/numpy_helper.h and src/headers/stdint.h` without breaking the build.

So the questions:
1) Is my intuition correct that there is more stuff being passed to the constructor than is really necessary?
2) Is there a cost to passing unnecessary junk? e.g. slower build time?
3) Is there a way to specify exactly the needed files instead of throwing the kitchen sink at it?

