How to install aws cli on a self-hosted azure DevOps Windows agent?
The best way to have AWS CLI tooling installed to your custom agent is by installing from Visual Studio Marketplace. AWS CLI is part of AWS Tools for Azure DevOps Service and Azure DevOps Server (the on-premise Azure DevOps) under the name of "AWS Tools for Microsoft Visual Studio Team Services".

Installing this AWS Tools will enable AWS tooling on CI (build pipeline) and CD (release pipeline).

This is the official landing page of AWS Tools in Visual Studio Marketplace: 
https://marketplace.visualstudio.com/items?itemName=AmazonWebServices.aws-vsts-tools

Official documentation is available at: https://docs.aws.amazon.com/vsts/index.html