How to Grow Rosemary – Harvesting. You can start to harvest your rosemary when it is 4 to 6 inches tall. However, do not prune it excessively until it is at least 8 to 10 inches tall. You can use rosemary directly out of your garden to add fresh seasoning to almost any dish.