An Illustrated Guide to SSH Forwarding
An Illustrated Guide to SSH Agent Forwarding



The Secure Shell is widely used to provide secure access to remote
systems, and everybody who uses it is familiar with routine password
access. This is the easiest to set up, is available by default, but
suffers from a number of limitations. These include both security
and usability issues, and we hope to cover them here.



 Table of Contents 

 Ordinary Password Authentication  
 Public Key Access 
 Public Key Access with Agent Support  
 Public Key Access with Agent Forwarding  
 How Key Challenges Work 
 Security Issues With Key Agents 
 Additional Resources 




In this paper, we'll present the various forms of authentication
available to the Secure Shell user and contrast the security
and usability tradeoffs of each. Then we'll add the extra functionality
of agent key forwarding, we hope to make the case that using ssh
public key access is a substantial win.



Note - This is not a tutorial on setup or configuration of Secure
Shell, but is an overview of technology which underlies this system.
We do, however, provide some pointers to information on several packages
which may guide the user in the setup process.





 Ordinary Password Authentication 


SSH supports access with a username and password, and this is little
more than an encrypted telnet. Access is, in fact, just like telnet,
with the normal username/password exchange.



We'll note that this exchange, and all others in this paper, assume
that an initial exchange of host keys has been
completed successfully. Though an important part of session security,
host validation is not material to the discussion of agent key
forwarding.



All examples start from a user on homepc (perhaps a Windows
workstation) connecting with PuTTY to a server running OpenSSH.
The particular details (program names, mainly) vary from implementation
to implementation, but the underlying protocol has been proven to
be highly interoperable.




  1 
  The user makes an initial TCP connection and
sends a username. We'll note that unlike
telnet, where the username is prompted as part of the connected data
stream (with no semantic meaning understood by telnet itself),
the username exchange is part of the ssh protocol itself.
 
   



  2 
  The ssh daemon on the server responds with a
demand for a password, and access to the system has not
yet been granted in any way.  
   



  3 
  The ssh client prompts the user for a password,
which is relayed through the encrypted connection to the server where
it is compared against the local user base.

   



   4 
   If the user's password matches the local credential,
access to the system is granted and a two-way communications path is
established, usually to a login shell.  
    




The main advantage of allowing password authentication is that
it's simple to set up — usually the default — and is easy to
understand. Systems which require access for many users from many varying
locations often permit password auth simply to reduce the administrative
burden and to maximize access.





The substantial downside is that by allowing a user to
enter a password, it means anybody is allowed to enter
a password. This opens the door to wholesale password guessing by users
or bots alike, and this has been an increasingly common method of system
compromise.



Unlike prior-generation ssh worms, which attempted just a few very common
passwords with common usernames, modern badware has a very extensive
dictionary of both usernames and passwords and has proven to be most
effective in penetrating even systems with "good" passwords. Only one
compromised account is required to gain entry to a system.



But even putting security issues aside, the other downside of password
authentication is that passwords must be remembered and entered
separately upon every login. For users with just one system to access,
this may not be such a burden, but users connecting to many systems
throughout the day may find repeated password entry tedious.



And having to remember a different password for every system is not
conducive to choosing strong passwords.



 Public Key Access 




To counteract the shortcomings of password authentication, ssh supports
public key access. A user creates a pair of public and private keys, and
installs the public key in his $HOME/.ssh/authorized_keys file
on the target server. This is nonsensitive information which need not
be guarded, but the other half — the private key — is protected
on the local machine by a (hopefully) strong passphrase.




A public key is a long string of bits encoded in ASCII, and it's stored
on one long line (though represented here on three continued lines for
readability). It includes a type (ssh-rsa, or others),
the key itself, and a comment:


$HOME/.ssh/authorized_keys
ssh-rsa AzAAB3NzaC1yc2EaaaabiWaaaieaX9AyNR7xWnW0eI3x2NGXrJ4gkQpK/EqpkveGCvvbM \
  oH84zqu3Us8jSaQD392JZAEAhGSoe0dWMBFm9Y41VGZYmncwkfTQPFH1P07vDw49aTAa2RJNFyV \
  QANZCbSocDeuT0Q7usuUj/v8h27+PqsUUl9XVQSDIhXBkWV+bJawc1c= Steve's key



This key must be installed on the target system — one time
— where it is used for subsequent remote access by the holder
of the private key.




  1 
  The user makes an initial connection and
sends a username along with a request to use a key.
 
 





  2 
  The ssh daemon on the server looks in the
user's authorized_keys file, constructs a challenge based
on the public key found there, and sends this challenge back to the
user's ssh client.

 





  3 
  The ssh client receives the key challenge. It
finds the user's private key on the local system, but it's protected
by an encrypting passphrase. An RSA key file is named id_rsa on
OpenSSH and SecureCRT, keyname.ppk on PuTTY. Other types
of keys (DSA, for instance) have similar name formats.

 





   4 
   The user is prompted for the passphrase to unlock
the private key. This example is from
PuTTY.

  





   5 
   ssh uses the private key to construct a key
response, and sends it to the waiting sshd on the other end of
the connection. It does not send the private key itself!

  





   6 
   sshd validates the key response, and if valid,
grants access to the system.

    





This process involves more steps behind the scenes, but the user
experience is mostly the same: you're prompted for a
passphrase rather than a password.
But, unlike setting up access to multiple computer systems
(each of which may have a different password), using public
key access means you type the same passphrase
no matter which system you're connecting to.





This has a substantial, but non-obvious, security benefit: since
you're now responsible for just one secret phrase instead of
many passwords, you type it more often. This makes you more
likely to remember it, and therefore pick a stronger passphrase
to protect the private key than you otherwise might.



Trying to remember many separate passwords for different remote
systems is difficult, and does not lend itself to picking strong
ones. Public key access solves this problem entirely.



We'll note that though public-key accounts can't generally be
cracked remotely, the mere installation of a public key on a
target system does not disable the use of passwords systemwide.
Instead, the server must be explicitly configured to allow only
public key encryption by the use of the PasswordAuthentication no
keywords in the sshd_config file.


 Public Key Access with Agent support 


Now that we've taken the leap into public key access, we'll take the next
step to enable agent support. In the previous section, the user's private
key was unlocked at every connection request: this is not functionally
different from typing a password, and though it's the same passphrase
every time (which makes it habitual), it nevertheless gets
tedious in the same manner.



Fortunately, the ssh suite provides a broker known as a "key agent" which
can hold and manage private keys on your workstations, and responding
to requests from remote systems to verify your keys. Agents provide
a tremendous productivity benefit, because once you've unlocked
your private key (one time when you launch the agent), subsequent
access works with the agent without prompting.



This works much like the key access seen previously, but with a twist.




  1 
  The user makes an initial connection and
sends a username along with a request to use a key.
 
 





  2 
  The ssh daemon on the server looks [1] in the
user's authorized_keys file, constructs a challenge based
on the key, and sends it [2] back to the user's ssh client.

 





  3 
  The ssh client receives the key challenge, and
forwards it to the waiting agent. The agent, rather than ssh itself,
opens the user's private key and discovers that it's protected by a
passphrase.

 





   4 
   The user is prompted for the passphrase to unlock
the private key. This example shows the prompt from
PuTTY's
pageant.

  





   5 
   The agent constructs the key response and
hands it back [1] to the ssh process, which sends it off [2]
to the sshd waiting on the
other end. Unlike the previous example, ssh never sees the private
key directly, only the key response.

  





   6 
   sshd validates the key response, and if valid,
grants access to the system. Note: the agent still retains the private
keys in memory, though it's not participating in the ongoing conversation.

    




As far as the user is concerned, this first exchange is little different
from key access shown in the previous section: the only difference is
which program prompts for the private key (ssh itself versus
the agent).



But where agent support shines is at the next connection request
made while the agent is still resident. Since it remembers the private
keys from the first time it was unlocked with the passphrase, it's able
to respond to the key challenge immediately without prompting.
The user sees an immediate, direct login without having to type anything.





Many users only unlock their private keys once in the morning when
they launch their ssh client and agent, and they don't have to enter
it again for the rest of the day because the resident agent is handling
all the key challenges. It's wonderfully convenient, as well
as secure.



It's very important to understand that private keys never
leave the agent: instead, the clients ask the agent to perform
a computation based on the key, and it's done in a way which
allows the agent to prove that it has the private key without
having to divulge the key itself. We discuss the challenge/response
in a later section.



Once agent support is enabled, all prompting has now been bypassed,
and one can consider performing scripted updates of remote systems.
This contrived example copies a .bashrc login config file to
each remote system, then checks for how much disk space is used
(via the df command):


# scripted update of several remote systems

for svr in server1 server2 server3 server4
do
	scp .bashrc $svr:~/   # copy up new .bashrc
	ssh $svr df           # ask about disk space
done



Without agent support, each server would require two prompts
(first for the copy, then for the remote command execution). With
agent support, there is no prompting at all.



However, these benefits only accrue to outbound connections made from
the local system to ssh servers elsewhere: once logged into a remote
server, connecting from there to yet a third server requires either
password access, or setting up the user's private key on the intermediate
system to pass to the third.



Having agent support on the local system is certainly an improvement,
but many of us working remotely often must copy files from one remote
system to another. Without installing and initializing an agent on
the first remote system, the scp operation will require a password
or passphrase every time. In a sense, this just pushes the tedium back
one link down the ssh chain.



Fortunately, there's a great solution which solves all these issues.


 Public Key Access with Agent Forwarding 


With our Key Agent in place, it's time to enable the final piece of
our puzzle: agent forwarding. In short, this allows
a chain of ssh connections to forward key challenges back to the
original agent, obviating the need for passwords or private keys
on any intermediate machines.





  1 
  This all starts with an already established
connection to the first server, with the agent now holding the
user's private key. The second server plays no part yet.
 
 






  2 
  The user launches the ssh client on
the first server with a request to connect to server2,
and this passes the username and a use-key request to the ssh
daemon (this could likewise be done with the scp secure
copy command as well)






  3 
  The ssh daemon consults the user's
authorized_keys file [1], constructs a key challenge from
the key, and sends it [2] back down the channel to the client
which made the request.

 





   4 
  
 This is where the magic occurs: the ssh client
on server receives the key challenge from the target system,
and it forwards [1] that challenge to the
sshd server on the same machine acting as a key agent.


sshd in turn relays [2] the
key challenge down the first connection to the original ssh client.
Once back on homepc, the ssh client takes the final step in
the relay process by handing the key challenge off [3] to the
resident agent, which knows about the user's private key.


  





   5 
 

The agent running on homepc constructs
the key response and hands it [1] back to the local ssh client,
which in turn passes it [2] down the channel to the sshd running
on server.


Since sshd is acting as a key agent, it forwards [3]
the key response off to the requesting ssh client, which sends it
[4] to the
waiting sshd on the target system (server2). This
forwarding action is all done automatically and near instantly.

  





   6 
   The ssh daemon on server2 validates
the key response, and if valid, grants access to the system.

 





This process can be repeated with even more links in the chain
(say, if the user wanted to ssh from server2 to server3),
and it all happens automatically. It supports the full suite of
ssh-related programs, such as ssh, scp (secure copy),
and sftp (secure FTP-like file transfer).





This does require the one-time
installation of the user's public — not private!
— keys on all the target machines, but this setup cost is
rapidly recouped by the added productivity provided. Those
using public keys with agent forwarding rarely go back.




 How Key Challenges Work 


One of the more clever aspects of the agent is how it can verify
a user's identity (or more precisely, possession of a private key) without
revealing that private key to anybody. This, like so many other
things in modern secure communications, uses public key encryption.



When a user wishes access to an ssh server, he presents his username
to the server with a request to set up a key session. This username
helps locate the list of public keys allowed access to that server:
typically it's found in the $HOME/.ssh/authorized_keys file.



The server creates a "challenge" which can only be answered by one
in possession of the corresponding private key: it creates and
remembers a large random number, then encrypts it with the user's
public key. This creates a buffer of binary data which is sent to
the user requesting access. To anybody without the private key, it's
just a pile of bits.





When the agent receives the challenge, it decrypts it with the private
key. If this key is the "other half" of the public key on the server,
the decryption will be successful, revealing the original random number
generated by the server. Only the holder of the private key could ever
extract this random number, so this constitutes proof that the user is
the holder of the private key.



The agent takes this random number, appends the SSH session ID (which
varies from connection to connection), and creates an MD5 hash value
of the resultant string: this result is sent back to the server as the
key response.



The server computes the same MD5 hash (random number + session ID)
and compares it with the key response from the agent: if they match,
the user must have been in possession of the private key, and access
is granted. If not, the next key in the list (of any) is tried in
succession until a valid key is found, or no more authorized keys
are available. At that point, access is denied.



Curiously, the actual random number is never exposed
in the client/agent exchange - it's sent encrypted to
the agent, and included in an MD5 hash from the agent.
It's likely that this is a security precaution designed to
make it harder to characterize the properties of the random
number generator on the server by looking at the the client/agent
exchange.



More information on MD5 hashes can be found in
An Illustrated
Guide to Cryptographic Hashes, also on this server.




 Security Issues With Key Agents 


One of the security benefits of agent forwarding is that the user's
private key never appears on remote systems or on the wire, even in
encrypted form. But the same agent protocol which shields the private
key may nevertheless expose a different vulnerability: agent hijacking.



Each ssh implementation has to provide a mechanism for clients to
request agent services, and on UNIX/Linux this is typically done
with a UNIX domain socket stored under the /tmp/ directory.
On our Linux system running OpenSSH, for instance, we find the file
/tmp/ssh-CXkd6094/agent.6094 associated with the SSH daemon
servicing a SecureCRT remote client.





This socket file is as heavily protected as the operating system allows
(restricted to just the user running the process, kept in a protected
subdirectory), but nothing can really prevent a root user from
accessing any file anywhere.



If a root user is able to convince his ssh client to use another
user's agent, root can impersonate that user on any remote system
which authorizes the victim user's public key. Of course, root can do
this on the local system as well, but he can do this directly
anyway without having to resort to ssh tricks.



Several environment variables are used to point a client to an agent,
but only SSH_AUTH_SOCK is required in order to use agent forwarding.
Setting this variable to a victim's agent socket allows full use of that
socket if the underlying file is readable. For root, it always
is.


# ls -l /tmp/ssh*      — look for somebody's agent socket
/tmp/ssh-CXkd6094:
total 24
srwxr-xr-x    1 steve    steve           0 Aug 30 08:46 agent.6094=

# export SSH_AUTH_SOCK=/tmp/ssh-CXkd6094/agent.6094

# ssh steve@remotesystem

remote$                  — Gotcha! Logged in as "steve" user on remote system!



One cannot tell just from looking at the socket information which remote
systems will accept the user's key, but it doesn't take too much detective
work to track it down. Running the ps command periodically on the
local system may show the user running ssh remotesystem,
and the netstat command may well point to the user's home base.



Furthermore, the user's $HOME/.ssh/known_hosts file contains a
list of machines which the user has a connection: though they may not
all be configured to trust the user's key, it's certainly a great
place to start looking. Modern versions (4.0 and later) of OpenSSH
can optionally hash the known_hosts file to forestall this.



There is no technical method which will prevent a root user from hijacking
an SSH agent socket if he has the ability to access it, so this suggests
that agent forwarding might not be such a good idea when the remote system
cannot be entirely trusted. All ssh clients provide a method to disable
agent forwarding.




 Additional Resources 


Up to this point, we've provided essentially no practical how-to
information on how to install or configure any particular SSH
implementation. Our feeling is that this information is covered
better elsewhere, and we're happy to provide some links here
to those other resources.






The Secure Shell: The Definitive Guide, 2 Ed
(O'Reilly & Associates).


This is clearly the standout book in its class: it covers Secure Shell
from A to Z, including many popular implementations. There is no
better comprehensive source for nearly all aspects of Secure Shell
Usage. A worthy addition to any bookshelf.



PuTTY


This is a very popular Open Source ssh client for Windows, and it's
notable for its economy (it will easily fit on a floppy disk).
The next resource provides extensive configuration guidance.



Unixwiz.net Tech Tip:
Secure Linux/UNIX access with PuTTY and OpenSSH


This is one of our own Tech Tips which is a hands on guide to
configurating the excellent open source
PuTTY
client for Windows.
Particular coverage was given to public key generation and
usage, with plenty of screenshots to guide the way.



Unixwiz.net Tech Tip: Building and configuring OpenSSH


Though this Tech Tip is mainly concerned with configuration of the
server on a UNIX/Linux platform, it also provides coverage of the
commercial
SecureCRT
Windows client from VanDyke Software (which we use ourselves). It
specifically details key generation and agent forwarding settings,
though briefly.



Unixwiz.net Tech Tip:
An Illustrated Guide to Cryptographic Hashes

 Though not central to using SSH Agent Forwarding, some coverage
cryptographic hashes may help understand the key challenge and
response mechanism. This Tech Tip provides a good overview of
crypto hashes in a similarly-illustrated format.




First Published: 2006-02-22




