OpenSSH

Overview

This page provides information about configuring and using SSH (or rather, the quasi-standard OpenSSH implementation).

References

http://www.securityfocus.com/infocus/1810

A very good introductory article about user identities.

http://www.securityfocus.com/infocus/1812

A very good introductory article about using ssh-agent

http://www.funtoo.org/Keychain

Website of the excellent keychain utility.

http://www.funtoo.org/OpenSSH_Key_Management,_Part_1

The first part of the article series "OpenSSH Key Management" by Gentoo creator Daniel Robbins. The second part also has details about using keychain. The third part deals with SSH agent forwarding.

Debian packages

The following Debian package needs to be installed:

ssh

This is a meta package which resolves to

openssh-server
openssh-client

Server configuration

Global configuration

The SSH daemon's configuration is located in

/etc/ssh/sshd_config

The default configuration is sensible, so generally I don't fiddle with this.

One thing that you may wish to change is the log level. The default log level is INFO, which logs quite a lot. For instance it logs when the Nagios service check disconnects without an attempt to authenticate. If this inflates the log file too much, you could say

LogLevel = ERROR

Note that there is no log level WARNING in-between INFO and ERROR. At the moment I am staying on INFO because I don't want to interfere with fail2ban.

For future reference: The (unencrypted) private and public host keys are located in

/etc/ssh/ssh_host_dsa_key
/etc/ssh/ssh_host_dsa_key.pub
/etc/ssh/ssh_host_rsa_key
/etc/ssh/ssh_host_rsa_key.pub

Per user configuration

To allow login by public/private key instead of password-based, those public keys that are allowed to login must be listed in

~/.ssh/authorized_keys

To quote from man sshd: "The content of the file is not highly sensitive, but the recommended permissions are read/write for the user, and not accessible by others.". My personal file therefore looks like this:

patrick@pelargir:~$ l .ssh
total 12
drwxr-xr-x 2 patrick patrick 4096 Jun 11 21:04 .
drwxr-xr-x 8 patrick patrick 4096 Jul 13 20:00 ..
-rw------- 1 patrick patrick  725 Nov 12  2009 authorized_keys
patrick@pelargir:~$ cat .ssh/authorized_keys 
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCuR259CxKjoSQd7cHy0IkXoUbmjqhTHkrSBX1NlZll0OluMVqZ/Pj6ntf4oqPL [...]

Note: See this HOWTO on the PGP page for a instructions on how to use GnuPG to generate an RSA key that is usable for SSH authentication.

Note: PuTTY generates public keys in PEM format. Use the following command to convert a public key from PEM format to OpenSSH format: ssh-keygen -f foo.pem -i. Other input formats are also possible and can be specified using the -m command line option.

Note: Sometimes you have an RSA private key from some other source that is not in OpenSSH format (starts with -----BEGIN RSA PRIVATE KEY-----). Use the following command to convert such an RSA private key to OpenSSH format: ssh-keygen -p -N "" -f foo.pem. Important: This will overwrite the original file! The command line option -p is actually used to change the passphrase, the conversion is merely a side-effect.

Client configuration

Location of configuration files

The client's configuration is located in these files:

• /etc/ssh_config : Default configuration for all users on the machine
• ~/.ssh/config : User-specific configuration

Identity/Public key authentication

Instead of providing a password for authentication, it is also possible to identify using an RSA or DSA public/private keypair. Basically it works like this:

• You generate a keypair
• You copy the public key to the remote machine and add it to the file ~/.ssh/authorized_keys in the target user's home directory
• You keep the private key on the local machine in the file ~/.ssh/id_rsa (if it's an RSA key) in your home directory
• You initiate an SSH session
• SSH asks you for the private key's passphrase
• If the passphrase is correct, the login will be successful.
• If you don't want to enter the passphrase for each new SSH session, you have to either use an unencrypted private key (BAD idea), or use ssh-agent (see appropriate section further down)
• Note: When logging in, the private key is never transmitted from local to remote machine! Instead the two machines work out a challenge that can only be "solved" by the local machine when it has access to the decrypted private key. When the user supplies the correct passphrase, the local machine gains access to the decrypted private key, can "solve" the challenge and therefore prove to the remote machine that the local user is indeed who he claims to be. To see all the stuff that happens behind the scenes during a login, use the command line option -v.

To create an RSA public/private keypair (public key is stored in ~/.ssh/id_rsa.pub, private key is stored in ~/.ssh/id_rsa):

ssh-keygen -t rsa

To view the fingerprint of an RSA key (using the public part of the keypair):

ssh-keygen -lf id_rsa.pub

Multiple identities

• If you want to allow multiple identities to login to a remote account, that account's authorized_keys file must contain a list of all those keys.
• If you want to have multiple identities on your local machine, each identity's private key must be placed in its own file. You then have to select the appropriate identity file using the -i command line option, or the identity file must be specified on a per-host basis in SSH's configuration file ~/.ssh/config. For instance, to use a different identity for each different user on a remote system, place the following snippet in ~/.ssh/config:

Host pelargir pelargir.herzbube.ch
# %r expands to the remote user name
IdentityFile ~/.ssh/%r.id_rsa

Multiple identities (roles) with which to connect to the same remote account

A real-life example that I use to distinguish between the administrator and user roles when working with Gitolite (see the GitServer page on this wiki):

Host gitolite-admin
HostName git.herzbube.ch
User gitolite3
IdentityFile ~/.ssh/admin.id_rsa
IdentitiesOnly yes

Host gitolite-user
HostName git.herzbube.ch
User gitolite3
IdentityFile ~/.ssh/patrick.id_rsa
IdentitiesOnly yes

What does this mean?

• We have defined two host aliases: One is called "gitolite-admin", the other "gitolite-user"
• Whenever we refer to one of these aliases in the future, SSH will use the options below the alias to perform the connection to the server
• If we want to perform administrative duties, we use the alias "gitolite-admin" which will cause SSH to use the "admin" identity
• If we want to do normal developer work (which hopefully will be most of the time) we use the alias "gitolite-user", which will cause SSH to use the "patrick" identity
• 'The IdentitiesOnly yes option is required to prevent ssh-agent from offering the wrong key file to SSH. Instead, ssh-agent is forced to only use the identity that is explicitly specified under the host alias. To understand the issue, let's have a look at the following scenario:
  • You have done some work under the normal user identity, which means that ssh-agent has now cached that identity's key
  • Now you want to perform admin work and try to clone the gitolite-admin.git repository - BOOM, cloning does not work and you get a mysterious message "Repository read access denied"
  • What happens here is this: Because ssh-agent has cached the normal user identity key, it offers that key to SSH every time a connection is about to be made. When SSH finds out that the key is allowed to login to the server's gitolite3 account, it happily accepts and uses the key. Once the connection has been established, SSH hands control over to the Gitolite software which finds out - too late - that the identity used does not have sufficient access rights to read the gitolite-admin.git repository

The problem here is conceptual: SSH only cares about authentication, not authorization - authorization is the domain of the application using SSH (Gitolite in this case). In other words: For SSH, every identity has the same "value": Either it is allowed to login (authenticate) to the remote side, or it is not. Gitolite, though, attaches an additional meaning to identities: Authorization, or access rights. The IdentitiesOnly option must therefore be used to ensure that the proper identity is used on every connection attempt, regardless of the state of the ssh-agent cache at connection time.

Using ssh-agent

If you make repeated connections to the same remote user/host, it quickly becomes tiresome to enter the same passphrase for accessing the authenticating RSA key over and over again. One solution would be to store the private key unprotected, i.e. without a passphrase. For obvious reasons, this is not a very good solution.

An alternative is to use the Authentication Agent ssh-agent. ssh-agent will cache a private RSA key once it has been decrypted so that it can be used repeatedly for SSH connections. It works like this:

• Start ssh-agent, e.g. when you log in.
  • This can be done automatically by adding the appropriate command to ~/.profile or some similar login-hook file.
  • On some systems, automatic startup of ssh-agent might already be configured in a similar way.
  • On Mac OS X, ssh-agent starts automatically when the first SSH connection is made.
• Use ssh-add to add the desired keys to ssh-agent
  • ssh-agent will prompt you for the passphrase (is this true? or is it ssh-add which does the prompting?)
  • On success, ssh-agent caches the decrypted key
• Make an SSH connection
  • When ssh would normally require the private key for its operation, it will first query ssh-agent whether or not it has the required key
  • ssh-agent never gives out the key itself. Instead, ssh tells ssh-agent to perform the required operation(s) on its behalf, and ssh-agent returns the operation's result back to ssh
• ssh-add -L lists all keys cached by ssh-agent
• ssh-add can also be used to remove keys from ssh-agent

Authenticating Agent Forwarding

Authenticating Agent Forwarding can be used to create an "authentication chain" that spans multiple systems. If you are logged in to a remote system A and from there would like to make another connection to remote system B, you can setup a forwarding mechanism so that ssh-agent on your local machine will be queried when you make the connection attempt to remote system B. In this way, the private authenticating RSA key never leaves your local machine!

A real-life application for this is when I'm on a remote machine and I want to clone a GitHub repository via SSH so that I have read/write access directly from that remote machine.

The forwarding mechanism is enabled like this:

ssh -A <host>

In the configuration file, the following enables forwarding:

ForwardAgent yes

Using keychain

The plain usage model of ssh-agent has two disadvantages:

• On every login a new instance of ssh-agent needs to be started and ssh-add needs to be run. Therefore if you login multiple times on a machine, you have to enter a passphrase multiple times: Once per login.
• Programs that are run by cron or otherwise run outside of a login shell do not have access to ssh-agent and thus cannot benefit from passwordless remote logins.

Enter keychain: This handy utility makes sure that a new instance of ssh-agent is started only on the first login after a reboot. Subsequent login sessions then reuse the same instance of ssh-agent. This means that you have to enter a passphrase only once. Because the single instance of ssh-agent is running in the background, detached from any login shell, even cron jobs can use that background instance of ssh-agent.

On a Debian system, install the package

keychain

Then enable the utility by adding the following line to ~/.bash_profile:

eval `keychain --eval --agents ssh id_rsa`

Discussion:

• The --agents option specifies which agents should be started by keychain. Currently only ssh-agent and gpg-agent are supported. If the option is omitted, keychain runs those agents that it detects on the system.
• id_rsa is the name of the file that stores the private key that should be added to ssh-agent. The file must be located in ~/.ssh. In the example above, the absolute path of the private key file is ~/.ssh/id_rsa. If more than one key should be added to ssh-agent, the files can be listed as separate arguments.
• The --eval option causes keychain to print lines to stdout that can be evaluated by the shell. keychain emits code that is specific for the shell that is referenced in the SHELL environment variable.
• Finally, the shell command eval evaluates the lines that keychain prints because of --eval

Refer to man keychain for additional options and documentation (including shell script snippets),

When keychain starts an instance of ssh-agent, it stores information about the process in a file named:

~/.keychain/<hostname>-sh

This file is a shell script snippet that can be sourced by scripts that are run by cron or inside other non-interactive shells. Example content:

pi@raspberrypi1:~$ cat .keychain/raspberrypi1-sh
SSH_AUTH_SOCK=/tmp/ssh-GuyeaCeibai/agent.12345; export SSH_AUTH_SOCK;
SSH_AGENT_PID=12345; export SSH_AGENT_PID;

The following shell script snippet can be added to any cron script that wants to use ssh-agent. The snippet also checks whether the private key id_rsa is present.

[ -z "$HOSTNAME" ] && HOSTNAME=`uname -n`
. $HOME/.keychain/$HOSTNAME-sh 2>/dev/null
ssh-add -l 2>/dev/null | grep -q id_rsa || exit 1

A final warning: Because ssh-agent keeps running even after you completely logout from the system, there is a certain additional risk in case someone gains unauthorized access to the system while the system is unattended: If an intruder manages to log in as root or as the user on whose behalf ssh-agent is running, the intruder at least can make use of the decrypted private keys that are cached by ssh-agent by logging in to remote machines. If sufficiently skilled, the intruder might even be able to extract the private key itself from the ssh-agent process. As a counter-measure you can add the --clear option to the keychain invocation in ~/.bash_profile. This causes keychain to clear the currently cached private key(s) from ssh-agent and ask again for the necessary passphrase(s). The idea is that keychain should treat every new login as a potential security breach. Obviously this defeats the convenience of entering the passphrase(s) only once per machine reboot, so --clear is really useful only on machines that are running unattended for long times.

Kudos: This excellent answer on the Unix StackExchange site pointed me towards keychain.

X11 Forwarding

The following command logs into a remote system and forwards X11 traffic from that system to the local system (which acts as X server):

ssh -X <host>

Note: It appears that X11 forwarding circumvents access control set up with xhost (at least this is the case on my Mac OS X box). It becomes clearer why this is the case if we look at how ssh sets the DISPLAY variable on the remote system: localhost:10.0. This means that the X11 connection is not made through a normal X11 network connection (which would be protected by xhost), but the connection is made through the SSH connection. To the local X server it appears that a locally launched program (= ssh) wants to display stuff.

X11 forwarding may be enabled by default by adding the following to the local config file (either globally or for specific hosts only):

ForwardX11 yes

Port Forwarding

Port forwarding (tunneling) is similar to X11 forwarding, but more generalized. The following command establishes port forwarding:

ssh -L 9110:mail.example.net:110 shell.example.net

From now on, all connections to localhost:9110 are routed via shell.example.net to port 110 at the destination host mail.example.net. Notes:

• mail.example.net sees the connection as coming from shell.example.net
• Traffic between shell.example.net and mail.example.net is not encrypted!!!
• All locally logged in users can see and use port 9110; this can be either useful or dangerous...
• Users that are logged in via a remote connection cannot see or use port 9110, unless the user establishing the forwarding tunnel has specified the -g option
• Ports below 1024 are privileged and can be used for forwarding only with root privileges; this makes sense if you think of a multi-user system where it is not desirable that one user can change an important port such as 25 (SMTP) or 53 (DNS) into a forwarding tunnel, without the other users noticing
• On the server side, port forwarding can be disallowed by specifying AllowTcpForwarding no in the server config file /etc/ssh/sshd_config. I have read somewhere, though, that this may be circumvented if someone is really determined

Port forwarding may also be enabled through the local config file:

Hostname shell.example.com
LocalForward 9110 mail.example.com:110

Note: Use RemoteForward to establish a "reverse-forwarding" connection, i.e. when you connect to a remote system, a port is forwarded from that remote system back to your local system.

Compression

Useful on a slow network connection: Data transferred over the SSH connection can be gzip compressed with the following command:

ssh -C <host>

Compression may also be enabled by adding the following to the local config file:

Compression yes
CompressionLevel <lvl>

Possible compression levels are 1 (fast, worst compression) to 9 (slow, best compression). The default level is 4.

Cipher selection

It is possible to select the Cipher that should be used for encrypting the SSH connection. Selecting a different cipher is probably most useful for SSH1 connections where the default cipher is the slow 3DES and it is desirable to select a faster cipher such as Blowfish.

The command to select a specific cipher is

ssh -c <cipher_spec> <host>

The cipher can be specified using one of the allowed keywords, or a comma-separated list of such keywords. The allowed keywords differ between SSH1 and SSH2. See the man page for details.

The cipher to use can also be specified in the configuration file (different options for SSH1 and SSH2):

Cipher <cipher>        # SSH1 connections
Ciphers cipher_spec>   # SSH 2 connections

Running remote shell commands

Instead of running an interactive shell on the remote system, it is also possible to run a single command:

ssh <host> ls -l

It is possible to pipe data to the remote command:

tar cf - <srcdir> | ssh <host> 'cat >dest.tar'

(the remote command in this case is enclosed in single quotes to prevent the local shell from evaluating the ">" character)

SSH on the Mac

Outgoing

Before macOS 10.12 (Sierra)

When a user starts his first SSH session to a remote machine, launchd automatically starts an ssh-agent process in the background, which from then on captures all private keys that are used to login to remote machines. If a private key is protected with a passphrase, a GUI dialog pops up where you have to enter the passphrase (instead of a command line prompt in Terminal.app). Optionally, you can allow the passphrase to be stored in your personal Mac OS X keychain, in which case you will not have to enter the passphrase in the future (the entry into the keychain is made so that ssh-agent, ssh-add and ssh always have access to the passphrase).

Note: If you change or delete a private key file, ssh-agent continues to cache the old key (no doubt with surprising results) until you remove the cached key using ssh-add -d.

X11.app is launched automatically when an SSH connection is made with X11 Forwarding.

macOS 10.12 (Sierra) and later

As of macOS 10.12 (Sierra), ssh-agent no longer auto-loads previously loaded ssh keys when you log in to your account. The previous behaviour can be restored by adding the following two lines to ~/.ssh/config for every section that contains an IdentityFile line.

UseKeychain yes
AddKeysToAgent yes

This solution comes from superuser.com.

macOS 10.13 (Ventura) and later

The version of OpenSSH shipped with macOS 10.13 (Ventura) and later by default no longer includes "ssh-rsa" in the list of signature algorithms that can be used for public key authentication. This can be a problem when trying to log in to machines that run an old version of OpenSSH. In my case, I noticed this specifically when logins to my RaspberryPi, which still runs Raspbian based on Debian jessie and OpenSSH_6.7p1, started to fail after I had upgraded macOS.

The issue can be diagnosed by passing three times the -v option to the ssh command:

ssh -vvv pi@raspberrypi1
[...]
debug1: send_pubkey_test: no mutual signature algorithm
[...]

If the above message is in the debug output, then this is an indicator for "ssh-rsa" not being accepted by the macOS version of OpenSSH.

For the moment, "ssh-rsa" is merely deprecated, but not yet unsupported. It can be enabled for specific hosts by adding the following two lines to ~/.ssh/config for every host that still needs "ssh-rsa".

PubkeyAcceptedKeyTypes=+ssh-rsa
HostKeyAlgorithms=+ssh-rsa

Important: Do NOT blindly add the two lines to a "Host *" section, only add them to hosts where it is needed!

Incoming

To allow SSH login to a Macintosh machine, go to "Preferences > Sharing" and enable "remote login" ("Entfernte Anmeldung" on German systems). You must provide admin privileges to do so.