How to set NACK on bad PEC in TWI used for SMBUS (ATMega8)
This is the main problem with trying to do SMBUS slave with PEC in software. And it is in fact mentioned in the standards as well as one of the reasons that PEC is not mandatory, but strongly suggested.

That means a master may not assume, by suggested best practise in the standard, that a slave sending an ACK in the last byte actually indicated the PEC was coded successfully. The only thing mandatory is that if a PEC is sent and it is invalid, the data must be discarded.

The only reliable way to do PEC and NACK as you want is if you have hardware support for either CRC8-CCITT (the PEC polynomial is the CCITT poly) or SMBUS, or preferably both.

Your chosen MCU cannot be made to do hardware I2C and also let you decide on the ACK bit last minute. That's just not something enough people want to put into every microcontroller. In 9 out of 10 cases the fact the slave expects another byte is enough to ACK. Or possibly 99 out of a 100 even.

You could hack it in hardware so that you can "NACK" just in time with an extra IO pin, but that would mean you need to make your CRC very efficient, or you'll be too late.