Why can anthracene, but not phenanthrene, take part in Diels–Alder reactions?
Have a look at the reaction mechanism of the Diels-Alder Reaction, e.g. at Wikipedia.
We have talked about this reaction before on this site, and concluded that a sufficient explanation is only possible with the help of molecular orbitals: How accurate is this polar mechanism for the Diels-Alder reaction?
Borrowing from Wikipedia,[1] here is the most important part of the mechanism, the HOMO of the diene and the dienophile:



Now the most important part of this picture is the symmetry of the HOMO/LUMO of the diene, since this is the subject of your question. We need either a HOMO or a LUMO of these symmetries and then could match a suitable dienophile. 

In anthracene we find the appropriate symmetry of the HOMO:

The reason for the regioselectivity has also been discussed before on this site: Diels Alder with Anthracene and Maleic anhydride.
We could also tune our reaction to be of inverse demand, as the LUMO has also appropriate symmetry:


In phenanthracene we do not have the appropriate orbital symmetry in the HOMO:

The LUMO is also not suitable for an inverse Diels-Alder reaction.


Reference and Notes


"FMO of Diels-Alder reaction" by Chrito23 - Own work. Licensed under CC BY-SA 3.0 via Commons.
Calculations on the DF-BP86/def2-SVP level of theory employing Gaussian09 rev. D. Contour values for the orbitals are at 0.05&nbsp;e/rBohr. Occupied orbitals are depicted in blue and orange and unoccupied orbitals are depicted in yellow and red. Phases are chosen at random.