 Most current algorithms do this, giving rise to the class of "generalized policy iteration" algorithms.Many "actor critic" methods belong to this category.The second issue can be corrected by allowing trajectories to contribute to any state-action pair in them.