Why verify padding at all? Whenever a ciphertext is decrypted using a block-cypher, we need to remove the padding. There are different ways to add padding, but they usually set the last byte of the last block to the number of padding bytes that were added and need to removed (e.g. we added 5 bytes, so we set the last byte to 0x05).

My question is: why do we need to verify padding? Why not read the last byte, remove that many bytes from the message, and be done with it?

Emitting a padding error sometimes catch integrity problems, but opens one to attacks like Padding Oracle, which is able to retrieve the entire plaintext in certain situations. It seems to me this is a terrible trade-off. What's the reasoning behind padding verification?
First of all, a more usual padding scheme would add 5 times the same byte 0x05 (in your example) so the check not just removes 5 bytes, but also checks that the 4 bytes before it have the same value. But let's assume your scheme (which is underspecified: what to put in the bytes before? Zeroes, or random values?) for now. 

What if you cannot remove that many bytes? Do you allow for final bytes that exceed the block length (so remove more than one block)? (TLS does do this.) Do you just discard the data? But that's also observable, probably. And if the length of the final output is somehow observable, you leak the final byte of the last block which could also be used for a padding oracle style attack.

In general, use a cryptographic MAC to protect your ciphertext, and verify it first. Emit those errors. Then padding errors should not occur at all.