What dangers are there with disturbed asbestos drywall?
OK, let's take about 10 steps back...


If you bought this home from a prior owner, did you hire a home inspector? Did that home inspector mention anything about the possibility of asbestos-bearing material? Asbestos in drywall is one I haven't heard before, but before its dangers became known it was virtually everywhere, so it wouldn't surprise me in the slightest to hear a house built in the 30s-50s has asbestos-bearing drywall (EDIT: It's not drywall as we know it, but "Transite", used as wallboard in places that needed to be especially fire-retardant, contained asbestos from its invention until the 1980s. The brand name survived, so there is Transite-labelled wallboard available today which does not contain asbestos). If you hired an inspector (it's a good investment; about $100-150 for an extra opinion of the safety and worthiness of the house), and he missed this, I would be looking at a lawsuit. You probably would not have bought this house knowing that every wall is potentially carcinogenic, and to fix the situation correctly is going to cost you, bigtime.
You say the plumbing people found out it was asbestos during the job. They can often find this out pretty quick, and any licensed plumber would, at that point, order masks on and get his crew the hell out of that house. This is a HUGE liability for the contractor; anyone who works for him that can link exposure at a job site to health problems can sue his company and him personally if he was their direct supervisor.
A permit is basically required by Canadian building codes whenever there is a change to one of the five main components of a house; foundation, structure, electrical, plumbing and HVAC. So, if the OP is from Canada I wouldn't doubt he'd need a permit to replace all the pipes in his house.
Your first mistake was in apparently hiring a plumbing specialist directly. If a company "doesn't do drywall", they are a plumbing specialist. There is nothing illegal about that, or about what they did; they assumed that you, in hiring them, were taking on the job of "general contractor" and that you would be managing all the various parts of the project (which includes patching drywall, AFTER having their plumbing job inspected). A better idea would have been to hire a general contractor for any work that involves having to get behind drywall. At least, as you admit, you and the plumber should have agreed on the scope of ALL work to be performed and detailed what necessary work the plumbing crew would not be doing, if any.
Asbestos is "ok" to have, as long as you leave it alone. The problem is when you disturb it; it's very brittle material, and breaking it up will release thousands of small fibers into the air, which if breathed in will cause a variety of lung diseases like asbestosis (the asbestos-related equivalent of miner's lung), mesothelioma (a formerly rare lung cancer linked directly to asbestos inhalation), and "vanilla" lung cancers similar to what can be caused by smoking. Asbestos in the drywall itself is very troubling; you risk exposure just by hanging a picture with a nail. If you were in the house at any point during the job, you almost certainly breathed some of it in.
The first step to resolving this problem is to document. Find all documentation involving the plumbing job. Hire a general contractor or asbestos specialist to take a look and confirm asbestos-bearing material in the drywall. If you had a home inspection, find the inspector's report and read through it.
Asbestos abatement is not cheap. In a case like yours, it's a "level 3" decon, basically amounting to a gut of your house back to the studs, with specialized negative-pressure air handling, decon showers, water misting etc to make sure that the fibers don't get out into the wider environment. The house will be completely uninhabitable until they're done, and it probably won't pass a HUD inspection after that. If your drywall has asbestos, I shudder to think what various types of insulation in this home contain; you may be left with an uninsulated outer skin and stud frame, and - are you sitting down? - it might well be cheaper to condemn, demolish and rebuild the building from the slab than to try to save any of it.
It MAY be possible and code-compliant to simply patch the drywall. However, depending on how rough the plumbing crew was, it may require disturbing more asbestos. A good patch job involves squaring the holes that were made, so a square piece of drywall can be cut to fit in the hole, then secured, filled, taped and plastered. The job will need to be done with proper breathing protection, and it will raise more asbestos dust that you will need to eliminate. You may not find a GC willing to touch this job without having ALL the asbestos out (see above), as it's a huge liability to anyone he brings in, and even after he's gone it'll be tough to confirm that the dust is all gone.
In all cases, I would hire a general contractor to do whatever work must be done; the general contractor will be able to tell you what disciplines he'll need to bring in and when, will schedule all of them for you, will resolve disputes, and keep tabs of the money being paid. At least, a GOOD GC will. There are guys who will take your money and run.