Smaller alternatives to sklearn that doesn't require scipy?
Did you check tinynumpy?
Anyway, I rarely found alternatives to famous packages (except scikit-image instead of opencv). What usually works for me is:

Slim the model as much as I can (e.g. weights quantization)
Check in the code which functions I use from each module. Once I have a list of them, I retrieve the corresponding python files and get rid of the rest
Try to split my process in multiple functions (e.g. one function to perform data processing, one function where to implement the model and make the inference)

The second point is crucial. In my experience, one rarely needs entire packages.
However, depending on the case, it could also be that AWS Lambda does not fit your needs.