Exclude Items from Flat Rate Shipping Calculation
This a bit of a complex setup and would be best handled by creating a new inline condition offering the ability to compare the order total of line items of a particular product type to some value. That's really what should be done.

However, this can actually be done with Rules. (Doesn't mean that it should, but it can!)

How would you do this? Well, you'll need a couple of components and a reaction rule. Go ahead and create a free shipping discount with Discounts and then modify the rule that gets created. You'll need to disable the Free shipping action and replace it with something that looks like this: Note that you can't skip the creation of the discount altogether because it's still needed to apply the discount itself (it's down at the bottom in the final rule).

{ "commerce_discount_rule_discount_free_shipping_for_75" : {
    "LABEL" : "Free shipping for $75",
    "PLUGIN" : "reaction rule",
    "OWNER" : "rules",
    "TAGS" : [ "Commerce Discount", "Order discount" ],
    "REQUIRES" : [ "rules", "commerce_discount" ],
    "ON" : { "commerce_discount_free_shipping" : [] },
    "DO" : [
      { "component_rules_amount_of_hats_on_order" : {
          "USING" : { "order" : [ "commerce-order" ] },
          "PROVIDE" : { "aount" : { "amount" : "Amount" } }
        }
      },
      { "component_rules_free_shipping_if_order_total_met" : { "order" : [ "commerce-order" ], "amount" : [ "amount" ] } }
    ]
  }
}


This rule has two custom actions on it.


The first one calculates the total amount of the hats on an order (I built this on Kickstart 2.x so change to suit your needs). The title of the rule isn't as important as is that you'll need a component to calculate a new total you'll compare against.
The second action will be the one that actually does the conditional application of free shipping.


Calculating a total on a subset of products

To calculate a total, we need to loop through all of the line items on an order, and then, if the line item's product is of a certain type, add it to a running total. Since Rules out of the box doesn't have if statements, we can emulate that using another custom component inside of the loop which will conditionally the current line_item being inspected add it's value to the running total if our product conditions are met. This could actually be anything, not just product type.

{ "rules_amount_of_hats_on_order" : {
    "LABEL" : "Amount of hats on order",
    "PLUGIN" : "action set",
    "OWNER" : "rules",
    "REQUIRES" : [ "rules" ],
    "USES VARIABLES" : {
      "order" : { "label" : "Order", "type" : "commerce_order" },
      "amount" : { "label" : "Amount", "type" : "decimal", "parameter" : false }
    },
    "ACTION SET" : [
      { "variable_add" : {
          "USING" : { "type" : "decimal", "value" : "0.0" },
          "PROVIDE" : { "variable_added" : { "running_total" : "Running total" } }
        }
      },
      { "LOOP" : {
          "USING" : { "list" : [ "order:commerce-line-items" ] },
          "ITEM" : { "list_item" : "Current list item" },
          "DO" : [
            { "component_rules_add_total_amount_of_line_item_to_total_if_product_is_of_ty" : {
                "USING" : { "lineitem" : [ "list-item" ], "total" : [ "running-total" ] },
                "PROVIDE" : { "total" : { "total" : "Total" } }
              }
            },
            { "data_set" : { "data" : [ "running-total" ], "value" : [ "total" ] } }
          ]
        }
      },
      { "data_set" : { "data" : [ "amount" ], "value" : [ "running-total" ] } }
    ],
    "PROVIDES VARIABLES" : [ "amount" ]
  }
}


Conditionally add a line item total to a running total

This is where most of the magic happens. This function is simple but could be expanded to do a lot of cool stuff. Currently, all it does is compare the product type to some value, but it could look at taxonomy terms, other fields... you name it! The gist of this rule is that it just does some comparison and then adds the line item value to the running total if it's met. It functions as an if statement. If the conditions are not met, the value doesn't increase.

{ "rules_add_total_amount_of_line_item_to_total_if_product_is_of_ty" : {
    "LABEL" : "Add total amount of line item to total if product is of type",
    "PLUGIN" : "rule",
    "OWNER" : "rules",
    "REQUIRES" : [ "rules" ],
    "USES VARIABLES" : {
      "lineitem" : { "label" : "Line item", "type" : "commerce_line_item" },
      "total" : { "label" : "Total", "type" : "decimal" }
    },
    "IF" : [
      { "entity_has_field" : { "entity" : [ "lineitem" ], "field" : "commerce_product" } },
      { "data_is" : {
          "data" : [ "lineitem:commerce-product:type" ],
          "op" : "IN",
          "value" : { "value" : { "hats" : "hats" } }
        }
      }
    ],
    "DO" : [
      { "data_calc" : {
          "USING" : {
            "input_1" : [ "total" ],
            "op" : "+",
            "input_2" : [ "lineitem:commerce-total:amount-decimal" ]
          },
          "PROVIDE" : { "result" : { "result2" : "Calculation result" } }
        }
      },
      { "data_set" : { "data" : [ "total" ], "value" : [ "result2" ] } }
    ],
    "PROVIDES VARIABLES" : [ "total" ]
  }
}


Conditionally applying the Free shipping

Finally, now that we've gotten the product-subset total amount, we can test against that and then apply our free shipping. Note that this rule will have the comparison of the product total and that's where you'd change the amount to compare against.

{ "rules_free_shipping_if_order_total_met" : {
    "LABEL" : "Free shipping if order total met",
    "PLUGIN" : "rule",
    "OWNER" : "rules",
    "REQUIRES" : [ "rules", "commerce_discount" ],
    "USES VARIABLES" : {
      "order" : { "label" : "Order", "type" : "commerce_order" },
      "amount" : { "label" : "Amount", "type" : "decimal" }
    },
    "IF" : [
      { "data_is" : { "data" : [ "amount" ], "op" : "\u003E", "value" : "74.99" } }
    ],
    "DO" : [
      { "commerce_discount_free_shipping_service" : {
          "entity" : [ "order" ],
          "commerce_discount" : "discount_free_shipping_for_75",
          "shipping_service" : "standard_shipping"
        }
      }
    ]
  }
}