What is the difference between regulatory and nonregulatory airspace?
Non-regulatory airspace is MOAs, Warning Areas, Alert Areas, and Controlled Firing Areas (source 1), (source 2). 

Other information should be located in the AIM Ch 3-4, Special Use Airspace. Specifically 3-4-1(c)

Basically non-regulatory areas mean that the FAA does not make regulatory enforcement in those areas due to special (military) operations. Usually the non-enforcement is only for participating aircraft, and generally recreational/commercial aircraft would not fall under any non-regulatory rules.

For example, in an active MOA with aircraft practicing low-altitude speed runs, the FAA would not enforce the 250 knot speed limit on participating aircraft. 

This is in the AIM, Ch 3 Section 4, Part 5(b) 


  Military pilots flying in an active MOA are exampted from the provisions of 14 CFR Section 91.303(c) and (d) which prohibits aerobatic flight within Class D and E surface areas and within Federal Airways. Additionally the Department of Defense has been issued an authorization to operate aircraft at indicated airspeeds in excess of 250 knots below 10,000 feet MSL within active MOAs..


For another example, consider warning areas, which are areas extending 3 miles from the coast. Since this area may include international waters (think of the area between Alaska and Russia), the FAA can't really regulate operations in all these areas. The US Navy also controls some of these areas and "shares" jurisdiction with the FAA.