This paper proposes a new mutated hybrid firefly (MHF) algorithm to improve overall system stability by optimizing the gain parameters for active and reactive power control of Distributed Energy Resources (DERs) in a microgrid system. A comprehensive optimization of entire DERs conditioning gain parameters are studied in terms of error function minimization. Photovoltaic (PV) system with battery energy storage (BESS) is proposed as one of the DERs to the microgrid. BESS system is used as an auxiliary power supply to improve the DC link stability of the voltage source converter (VSC). A second order PLL based VSC is considered for grid synchronization for this DER. Diesel engine (DG) with Pade approximation based governor control is incorporated in the proposed microgrid. A low voltage weak AC microgrid is considered for confirmation of increased stability limits through computer simulations using MATLAB editor. The stability limits are studied in terms of DERs dynamic performance. Both active and reactive power dynamic relations and 2nd order PLL feedback system of the PV-VSC system are further studied for validation of stability. Various case observations are presented including both microgrid operational and solar irradiation deviations for performance evaluation of the DERs. (C) 2016 Elsevier Ltd. All rights reserved.