Can I use MAC as CTR mode nonce, if my messages are non-repeating? I design encryption for a network protocol with restrictively low MTU. I'm aiming at 128 bit strength; AES-128-CCM would add 128 nonce bits and 128 MAC bits to each packet. Such MTU reduction by 32 bytes is not acceptable and I must come up with a solution which adds smaller overhead.

The protocol messages do not repeat between re-keying, because they contain large enough sequence number (I can specify how often to re-key). 

Question: Can I use the packet MAC as the nonce for the CTR mode encryption?

AFAIK the only requirement for the "counter" value is that it does not repeat for the same key, and that is guaranteed by message uniqueness (assuming MACs would not collide). Such "reuse" would save me 16 nonce bytes in each packet.
Yes, but to use a known algorithm that specifies precisely that, take a look at the SIV mode of operation . "SIV" stands for Synthetic IV. IV is the initialization vector, which is the nonce (or the nonce and initial counter value, depending on the definitions).

Unsurprisingly it relies on CTR mode for confidentiality and CMAC for the authentication tag that doubles as IV - much like CCM.

The problem with CCM is that you would have a chicken/egg problem as the MAC is calculated at the end of the protocol. Although you could implement the protocol differently, I'd recommend to use a tried and tested mode instead.



Besides that, CCM mode is - at least in my opinion - a horrible mode to implement.

SIV does have the problem that it is a full two pass protocol; you will have to go over each packet twice to encrypt: one pass for creating the SIV and on pass to encrypt. Decryption is easier as you already have the IV ready, so you can at least combine the two passes.