What was happening in Madagascar during 1971-1996?
I think I have found the explenation for this. If you look at the wikipdedia page of the economy of Madagascar, it states the following:

The standard of living of the Malagasy population has been declining dramatically over the past 25 years. The country has gone from being a net exporter of agricultural products in the 1960s to a net importer since 1971. Inappropriate traditional agricultural methods cause soil to erode and soil quality to decline, and the basis of survival for Madagascar’s people is under serious threat.

The source of this claim is Deutsche Gesellschaft für Internationale Zusammenarbeit. 

Now, if you look at the world bank figures on agriculture in Madagascar, you can see this declining trend. For example the Agriculture value added per worker (constant 2005 US$) declines between 1980 and 2014 from 250.1 to 175.8. 

Since Madagascar is a country which relies heavily on agriculture (about a quarter of its GDP stems from agriculture), this declining trend influences the whole of this economy.