Base stations equipped with resources to harvest renewable energy are not only environment-friendly but can also reduce the grid energy consumed, thus bringing cost savings for the cellular network operators. Intelligent management of the harvested energy can further increase the cost savings. Such management of energy savings has to be carefully coupled with managing the quality of service so as to ensure customer satisfaction. In such a process, there is a trade-off between the energy drawn from grid and the quality of service. Unlike prior studies which mainly focus on network energy minimization, this paper proposes a framework for jointly managing the grid energy savings and the quality of service (in terms of the network latency), which is achieved by downlink power control and user association reconfiguration. We use a real BS deployment scenario from London, U.K., to show the performance of our proposed framework and compare it against existing benchmarks. We show that the proposed framework can lead to around 60% grid energy savings as well as better network latency performance than the traditionally used scheme.