What happens if you receive a block with an unknown previous hash in the header I searched this question and couldn't find an answer to the following scenario.

Block 4440 gets mined at 10:00 AM
Block 4441 gets mined  at 10:01 AM

Lets assume I receive Block 4441 first and 4440 later. What happens? When I received 4441 I didnt know the block with the previous hash and I wasn't able to add 4441 to my chain. 

Do I store these kinds of blocks with an unknown previous header in a pool and check periodically if I received the previous block?

Assuming we do this, do you ever remove old blocks from this pool?

If we save "invalid" blocks, how do we prevent people from submitting thousands of invalid blocks and filling up our pool? I assume we check if the difficulty was met but what happens if block 4440 lowered the required difficulty?
Blocks with an unknown parent block are stored in the orphan block pool. They will stay there until the parent block is received and then linked to the existing chain. 

As far as I know, there's no other condition for the block to be removed from the pool. If the particular parent is never received, they will remain there forever.

The current difficulty should be enough to prevent people from spamming this pool with orphans, because that would require a lot a processing power and a huge waste of money. A block has to be valid to be stored in this pool, block hash has to meet target and so on. If that is not the case, the block is discarded and will not spam the orphan pool.

See Mastering Bitcoin, Chapter 10, Section Assembling and Selecting Chains of Blocks for detailed explanation.