Distortion is introduced when speaker is connected to the output of the amplifier
The OP484 is an operational amplifier, not a power amplifier.  Your speaker is simply too much of a load for that IC to drive.

You could get around this by including a push/pull transistor stage at the output of the opamp (inside the feedback loop.)  But, since you have already ordered an LM386, you could just wait until you get your power amplifier.



The LM386 is (despite its schematic symbol) not an opamp.  The circuit you put together for the OP484 won't work for the LM386.  The suggested minimum circuit for the LM386 is very different.  Follow the suggestions in the datasheet.  Pay special attention to the gain setting - it is nothing like how you set the gain on an opamp.