What happens when the buffer pool size is insufficient in mysql
If MySQL properly started and InnoDB initialized with the configured buffer pool size then there is no reason for any error (barring some bug of course). The buffer pool is just that - a memory area used to buffer your IO. When there is not enough of it to keep all the active data, the server just reads them from disk.

If you had for example 100GB table then many servers won't have enough RAM to contain it and still you may execute a query running full table scan.

The performance will be worse (probably substantially, 1-2 orders of magnitude or more) than in case there were enough memory.