Should I be worried about the weight of french drain gravel on my foundation?
Sure, everything affects the load on your footing. However, eliminating moisture (water) on your footing is more important. 

There are three types of loads  that affect the displacement of your footing: 1) Dead Loads, 2) Live Loads, and 3) Superimposed Loads. 

1) Dead Loads include the weight of the footing, building loads on the footing, and soil on and around the footing. Depending on the “angle of repose” of your soil conditions, all the soil or gravel next to your footing can add a load to the existing soil bearing. 

Gravel can weigh about 90 lbs. per cubic foot (dry) and soil is about 100 lbs. per cubic foot (dry and depending on the type) and water is about 62 lbs. per cubic foot. Wet soil and water soaked gravel is considerably heavier. 

As you can see, gravel and soil has similar weight characteristics. So, when you replace the soil with a 4” perf pipe and gravel, the weight is probably less, due to the void in the pipe. 

2) Live Loads of people, benches, landscaping, etc. also adds weight and should be considered. 

3) Superimposed Loads such as a parked car next to the footing also adds weight and should be eliminated from the area by the footing. 

Summary:

Many things contribute to loading weight to the soil under the footing and can contribute to settlement. Just because the soil or gravel is not directly on top of the footing, it can still add “load” to the soil under the footing due to the angle of repose of the soil. A safe rule of thumb is anything (including Live Loads and Superimposed Loads) within a 45 degree angle from the edge of the footing will contribute “load” and cause settlement. 

BTW, don’t install the gravel below the existing footing within the angle of repose. Otherwise you could undermine the bearing of the footing.