ZigBee end-device could not connect to Coordinator
Some things to look at to improve the layout:


If this is only a 2 layer board, you may have some grounding issues around the radio.  Traces on the bottom layer run directly under the RFN and RFP pins.  They also cut up the ground plan quite a bit on top and bottom.  Consider going to a 4 layer board to maintain a continuous ground plane across the board.  Of course you'll need to maintain the cutout around the antenna.  
With #1, place ground vias as close to the MCU/Radio pins as possible.  MCU pins 7 and 10 are lacking vias.
Also consider adding more vias to the center pad of the MCU/Radio IC to minimize ground inductance.
How did you calculate 50 ohm impedance for the RF trace?  Take note that there is a difference between microstrip and coplanar waveguide.
Although not a layout item and may not be related, the datasheet recommends pulling CLKI pin low if not used.


You can refer to the eval kit for this radio for how they implemented the above.

ATmega256RFR2 Xplained Pro Eval Kit

Although the above recommendations should improve performance, I would expect you should still be able to communicate with another device over the air at the short distances you have mentioned.  Check basic operation with your board to ensure it is doing what you intend:


Power supplies are up and stable
Crystals are operating at their proper frequencies
No shorts on the RF output pins, balun, etc.
Is the balun installed correctly, with proper orientation
The link to the balun that you provide is different from the PN in the schematic.  Baluns are designed to match the output impedance of the radio to 50 ohms, so not having the proper balun matching would reduce performance.
Try multiple boards to ensure its not an assembly issue.


Once you get your device communicating, I suggest you follow the recommendations from your previous post about contracting an RF engineer to dial your system in for optimal performance.  You can check the RF match of the antenna, measure your RF output power to ensure expected output, tune your RF frequency to ensure expected output, etc.  All of which requires expensive equipment.