Is the Fruit of Grisaia an actual fruit?
According to this interview with Front Wing Producer, President, and CEO, Ryuuichirou Yamakawa:


  山川　絵画の画法でモノトーンで書く手法を「グリザイユ」というのですが、これを少し、いじって『グリザイアの果実』つまり「灰色の果実」という意味が込められています


The word "grisaia" is a corruption of the French term used in art, "grisaille," a method of painting in monochrome or near-monochrome, typically to resemble the appearance of marble sculptures. So literally it's meant to be "the gray-colored fruit" or something like that.

The fruits refer "guilt" the heroines bear, as described by the story synopsis of the game:


  ──その学園は、少女達の果樹園だった。
  外敵から隔離された学園にやってきたのは、生きる目的をなくした一人の少年。
  守るべき物を見失い、後悔と贖罪のみに費やされる人生の中で、その少年に残されたのは首に繋がれた太い鎖と、野良犬にも劣る安い命。
  そして少年は、その学園で少女達と出会い、新たな希望を見つけ出す。
  
  ──その少女は、生まれてきたことが既に間違いだった
  逆らった罪──
  ──生きながらの死
  誰も守ってなんかくれない──
  ──そして生き残った罰。
  
  そこは、少女達の果樹園。
  彼女達は、後悔の樹に実った懺悔の果実。そんな少女達に、俺はいったい何が出来る…？
  それは、一人の少年が夢見た永遠の希望──
  
  That academy – used to be the orchard of young girls.
  A lonely boy, who lost his purpose to live, came to that isolated-from-the-foreign-enemies academy. 
  Lose sight of things that should be protected, he spends his life in living day by day with regret and atonement. His life is cheaper than a stray dog when that heavy cangue still around his neck.
  And then at that academy, the boy finds out his new hope in encountering with these girls.
  
  ──That girl, who being born was already a mistake.
  Who defied the sin──
  ──Who died though she’s living.
  Who no one will protect── [Note: translation corrected]
  ──and who being punished for surviving.
  
  That place is the girls’ orchard.
  The trees of regret bear the fruit of repentance of those girls. What on earth can I do for these girls...?
  That is the eternal hope which the lonely boy dreamt...


Each heroine is represented by a fruit: 


Yumiko = grape
Suou = cherry
Michiru = lemon
Irisu = strawberry
Sachi = apple