Mud bricks with shale
Your idea has merit. Shale is classified by geologists as being a mudstone. It is composed of silt and clay sized minerals.

According to this geology website, shales can be crushed to produce a clay and then mixed with water. These days, items that were once made from natural clay, such as terracotta pots and tiles for roofs, are now made from crushed shale.  



Extra Information

One thing to be mindful of is that clays expand when wet and contract when dry. Anything you make from it, such as bricks, that isn't baked or protected from water by a sealed coating will expand and contract according to it moisture content.

Another application which you may want to consider is to supply crushed shale as a garden mulch or other such landscaping product.