Spread a foundation layer of "mortar" and set the crackers into it.
To make walls, set crackers vertically, gluing them to each other with the "mortar".
Glue small crackers onto the larger ones to make doors, windows and shutters then cement graham crackers in place as a sloped roof.