Using git over ssh from localhost to my Shared Hosting
Perhaps you are looking for

git clone ssh://username@63.23.133.162:portnumber/~/public_html/drupal


Note the ~ which is indicating that the public_html resides in your home directory.