How is this crumbly crystal-rich rock formed?
Even rocks that have lasted for billions of years decompose when exposed to Earth's weather.

What you have is indeed granite, but a somewhat decomposed granite, which is treated by geologists differently from the intact, solid stuff.  The "crumbly bits" have their own 
name: grus.

Granite is made of crystals of different minerals  (mostly feldspar and quartz) that grew into each other as the parent magma cooled. The final rock is solid because of the irregular way the crystals fit into each other.

Granite weathers in several different ways. Cracks can form from hot and cold cycles. However, chemical weathering is the principal way granite becomes crumbly.  Rain frequently picks up some acidity from the atmosphere, and infiltrates the granite. This acidic rain reacts with the feldspar and mica to produce clay minerals, primarily kaolinite. 

This in turn makes it easier for water to get in the granite, and combined with heat cycles the process accelerates.
As a result, the joints between the crystals loosen up, producing rock in the same condition as your rock.