Which altimeter setting is used to define airspace boundaries?
Airspace boundaries are not defined by altimeter settings or Indicated Altitude. Like the field elevation, they are at True Altitude. This is defined as the actual height above the 19-year average level of the sea surface. Some boundaries are defined by the height above the surface (field elevation) which is in turn at True Altitude. The MSL vertical dimensions of the boundaries do not change based on barometric pressure or altimeter settings. They are relatively constant.
Your Indicated Altitude shown on your Altimeter may be different from the True Altitude due to barometric pressure changes, erroneous settings, or atmospheric anomalies. This does not change the boundary dimensions.
For further clarity, look at how the FAA Pilot’s Handbook of Aeronautical Knowledge defines these terms:

Types of Altitude
Altitude in itself is a relevant term only when it is specifically stated to which type of altitude a pilot is referring. Normally when the term “altitude” is used, it is referring to altitude above sea level since this is the altitude which is used to depict obstacles and airspace, as well as to separate air traffic.
Altitude is vertical distance above some point or level used as a reference. There are as many kinds of altitude as there are reference levels from which altitude is measured, and each may be used for specific reasons. Pilots are mainly concerned with five types of altitudes:

Indicated altitude—read directly from the altimeter (uncorrected) when it is set to the current altimeter setting.
True altitude—the vertical distance of the aircraft above sea level—the actual altitude. It is often expressed as feet above mean sea level (MSL). Airport, terrain, and obstacle elevations on aeronautical charts are true altitudes.



Mean sea level.
The average height of the surface of the sea at a particular location for all stages of the tide over a 19-year period.

If you are on flight following, or an IFR flight plan, it is up to the controller to clear you through an airspace or hand you off to the controller with jurisdiction. If you are not on flight following, or an IFR flight plan, don’t fly so close to the boundary without giving the local tower a courtesy call to “request” transiting their airspace at your desired altitude. It is not a bad idea to do so even if you are well above their airspace. They may know about conflicting traffic of which you are not aware.