Is booth multiplication algorithm for multiplying 2 positive numbers?
I think it is because you are using 3 bits, and for 3 bits there is no complement of 5.
Try it with 4 bits. I just tried it and it seems to work for me.  

Here's an example:

A = 0101 0000 0
S = 1011 0000 0
P = 0000 0100 0


1st step

Last 2 digits of P are 00 so we arithmetic right shift giving:

P = 0000 0010 0


2nd step

Last 2 digits of P are 00 so we right shift giving:

P = 0000 0001 0


3rd step 

Last 2 digits of P are 10 so we add P to S giving:

P = 1011 0000 1


Then right shift giving:
P = 1101 1000 1 (note the MSB replication so 1 is shifted in)

4th step 

Last 2 digits of P are 01 so we add P to A giving:

P = 0010 1000 1


Then right shift giving:
    P = 0001 0100 0

Then finally remove the LSB giving:

0001 0100 = 20
Which is the product of 4 * 5.