Is there a hook for custom line item before rules price calculation?
As Clive mentioned, there's hook_commerce_product_calculate_sell_price_line_item_alter() that lets you manipulate the line item prior to the sell price calculation process executing on it. However, whatever unit price you set via that hook may be overridden by Rules that you weren't intending to apply to those line items. In my opinion, the best thing to do would be to write a custom Rules action to encapsulate your pricing and call it in a sell price calculation rule.