rtmp: fix buffer overflows in ff_amf_tag_contents()

A negative `size' will bypass FFMIN().  In the subsequent memcpy() call,
`size' will be considered as a large positive value, leading to a buffer
overflow.

Change the type of `size' to unsigned int to avoid buffer overflow, and
simplify overflow checks accordingly. Also change a literal buffer
size to use sizeof, and limit the amount of data copied in another
memcpy call as well.

Signed-off-by: Xi Wang <xi.wang@gmail.com>
Signed-off-by: Martin Storsjö <martin@martin.st>
