Removed Outer Brick on an Old Fireplace. Need Advice on how to re-finish
Any fireplace than can be used as a wood burning fireplace needs to be built to certain specifications set forth by the building code. The brick you are referring to is most likely backer brick as suggested by Karas but it will be backer for the smoke chamber. The smoke chamber must have a few cracks in it if you seen smoke infiltration coming through. FWIW, it will not need to be fire brick, that is only needed in the firebox only. Anything above, behind and below the firebox can be standard brick. the walls of the smoke chamber and firebox, for the most part the whole chimney, must be 8" of solid masonry everywhere. 

Some of that backer may be able to be removed, but you must maintain 8" of solid, tight joint brick. After that the wood framing or any combustible material must maintain a 2" air space to avoid fires. 

As you can see, when the fireplace was built, that wasn't too much of a worry. I can only recite what the code requires now....

To get past some of this, metal framing can be used, but the sheetrock must be 2" away from the brick at the smoke chamber and firebox. 

If the fireplace was not ever a wood burning fireplace and was made originally only as a gas fireplace, these issues would not be a concern.