How can I create a loop to cycle thru each category and the the entries within each category?
First you loop through the categories then use the category to loop through all the entries related to that category.

{% for category in craft.categories.group('categoryGroup') %}
  {% for post in craft.entries.section('posts').relatedTo(category) %}
    {# your markup #}
  {% endfor %}
{% endfor %}