What fundamental differences separate low-budget airlines from traditional airlines?
Generally speaking:


They operate a limited number of airframes if not a single one (for example Ryanair operates 420 737-800's and only a single 737-700 for chartering). This allows them to buy planes in bulk as well as cut down on spare parts reserves, pilot proficiency training in multiple airframes and overall maintenance costs by only needing to hire mechanics proficient in a single airframe. They can also easily substitute an aircraft on a given route as they are all configured the same. 
By having seats that don't recline and sometimes no first class they can really pack people in. Those extra customers make a big difference. There has even been some buzz about packing more people in with "standing" seats but I have not seen anything certified on that front as of yet. 
In my experience they tend to have less lenient refund and change policies. For example Spirit charges a $90 cancelation fee and most tickets are around that price. 
The meals tend to be pre-packed and may have a longer shelf life than the hot food served on other airplanes. This means they can buy it in bulk (read cheap), in advance and store it, and there is no need to coordinate the delivery like there is with hot food. Cold food can hang out at the gate waiting for the plane. 
There has been a big move to use mobile devices for a lot of classic functions like check in and boarding passes. This allows them to not need checking counter employees or curb side employees etc. This labor cost is a huge savings. In many cases the flight attendants may also be the  boarding/gate personnel.     
Some of them also avoid major airports and may elect to fly out of smaller regional airports. This helps keep fees down and some satellite airports may be substantially less busy allowing them to get in and out faster.  


As per @Jamiec's comments 


Shorter turn around times 


No seat back pockets so nothing to clear out. 
No pillows and blankets to pick up 
Cabin crew may do the majority of cleaning 



As @757toga notes some of them also may keep pay low or act as a jumping off point for young pilots to get their type certs and build hours.  

Related questions:

How can Ryanair offer flights for a penny?
What's the typical cost (and its breakdown) for a long-haul commercial flight?