SSH: RSA keys, and ssh-agent for SSH keys and their passwords management

During keyring configuration for the Nextcloud client (see the Linux: the Nextcloud client, qtkeychain and the “The name org.freedesktop.secrets was not provided by any .service files” error post) – I decided to clean up the mess in my SSH keys, as I have a lot of them and sometimes authentication became just pain.

In general to make this simpler one can use system-wide storage like gnome-keyring or KeeyPassXC, but we will speak about them in the next post.

Today, let’s discuss ssh-agent and how to use it to manage password-protected RSA keys for SSH authentication without such a backends.

Examples below performed on an Arch Linux installation with some additional tests Manjaro Linux with Budgie DE.

Contents

`ssh-agent`

ssh-agent is intended to manage a user’s SSH keys and their passwords to avoid the necessity to enter a key’s password each time you need to log in a remote host using such a key for your authentication.

Running the agent

Just perform:

[simterm]

$ ssh-agent
SSH_AUTH_SOCK=/tmp/ssh-dMDE5mED77tM/agent.436347; export SSH_AUTH_SOCK;
SSH_AGENT_PID=436348; export SSH_AGENT_PID;
echo Agent pid 436348;

[/simterm]

For clients, such as ssh-client or git, they need to know the following variables:

SSH_AGENT_PID: a started ssh-agent PID, that will be sued for example to kill it with ssh-agent -k
SSH_AUTH_SOCK: a path to a UNIX socket file which will be used to communicate to the ssh-agent from clients (ssh, git, etc)

To run an agent without displaying these variables and to apply them – do the next:

[simterm]

$ eval $(ssh-agent) > /dev/null

[/simterm]

There are few various ways to run the agent, we will take a closer look at the Running ssh-agent with multitype terminals part.

Examples

Let’s take an overview of ssh-agent basic usage.

A key generation

Create a new key:

[simterm]

$ ssh-keygen -t rsa -b 2048 -f /home/setevoy/.ssh/test-key -C "Testing key" -P pass 
Generating public/private rsa key pair. 
Your identification has been saved in /home/setevoy/.ssh/test-key. 
Your public key has been saved in /home/setevoy/.ssh/test-key.pub. 
The key fingerprint is: 
SHA256:pTyrGtk1hnNHB6b8ilp5jRe1+K4KrLHg50yUGilApLY Testing key 
The key's randomart image is: 
+---[RSA 2048]----+ 
|.o        o      | 
|o      . o .     | 
|o.      o o o    | 
|o .. . o = + .   | 
|.Eo o o S = .    | 
| . + + B O o     | 
|  o = B = o .    | 
| . +.B + . .     | 
|  .oB.. .....    | 
+----[SHA256]-----+

[/simterm]

Options here:

-t: type, RSA
-b: a key’s length in bits (by default 3072 for RSA)
-f: a path to the key’s file (by default ~/.ssh/id_rsa)
-C: a comment for the key (by default username@hostname)
-P: a key’s password

Checking SSH-key’s password

To check a password for a key you can use thessh-keygen with -y to display information about this key, and this will ask you to enter this key’s password:

[simterm]

$ ssh-keygen -y -f /home/setevoy/.ssh/test-key
Enter passphrase:
ssh-rsa AAAAB***gud2vedL/V Testing key

[/simterm]

`ssh-copy-id` – copy a key to a remote host

You can copy a key manually, by getting its public part from the test-key.pub file:

[simterm]

$ cat .ssh/test-key.pub
ssh-rsa AAAAB***gud2vedL/V Testing key

[/simterm]

And by adding it to the ~/.ssh/authorized_keys on a target host.

Another way, the recommended one, is to use an ssh-copy-id utility which will do the same but also will keep an eye on folders/files permissions – the most frequent problem during SSH RSA-based authentification:

[simterm]

$ ssh-copy-id -i /home/setevoy/.ssh/test-key [email protected]
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/setevoy/.ssh/test-key.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
[email protected]'s password:

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh '[email protected]'"
and check to make sure that only the key(s) you wanted were added.

[/simterm]

Now you can log in using this key:

[simterm]

$ ssh [email protected] -i .ssh/test-key
Enter passphrase for key '.ssh/test-key':
Linux rtfm-do-production 4.9.0-8-amd64 #1 SMP Debian 4.9.144-3.1 (2019-02-19) x86_64
...
setevoy@rtfm-do-production:~$

[/simterm]

`ssh-add`

Okay, so now we do have a password-protected RSA key for SSH authentication.

But during each SSH-login, you’ll have to enter its password again and again and this will be a real pain when using a lot of connections and keys.

To avoid this issue – add a key to the ssh-agent using ssh-add.

Check if it is running:

[simterm]

$ ps aux | grep ssh-agent
setevoy     1322  0.0  0.0   5796   456 ?        Ss   Nov30   0:00 ssh-agent -s
setevoy     1324  0.0  0.0   5796  2160 ?        Ss   Nov30   0:00 ssh-agent -s
...

[/simterm]

Could not open a connection to your authentication agent

The most common problem is when ssh-add is not able to connect to an agent:

[simterm]

$ ssh-add
Could not open a connection to your authentication agent.

[/simterm]

At first – check if it’s PID is set from the SSH_AGENT_PID, or by checking the $SSH_AUTH_SOCK variable as all communication is gone via the socket-file specified by this variable:

[simterm]

$ test -z $SSH_AGENT_PID; echo $?
0

[/simterm]

Here is it empty, because thessh-agent was started in another terminal instance (we will speak soon how to handle it).

For now – kill all already running instances:

[simterm]

$ killall ssh-agent

[/simterm]

And run an agent’s instance anew:

[simterm]

$ eval $(ssh-agent -s)
Agent pid 452333

[/simterm]

We are using the-s option as not everybody will do the steps above from the exactly bash shell and eval to apply the strings from the agent’s output (export SSH_AUTH_SOCK).

Check again:

[simterm]

$ test -z $SSH_AGENT_PID; echo $?
1

[/simterm]

And ssh-add:

[simterm]

$ ssh-add -l
The agent has no identities.

[/simterm]

All done here.

Adding a key

Run:

[simterm]

$ ssh-add /home/setevoy/.ssh/test-key
Enter passphrase for /home/setevoy/.ssh/test-key:
Identity added: /home/setevoy/.ssh/test-key (Testing key)

[/simterm]

Checking keys

Use the -l option to check which keys are already loaded to an agent’s instance:

[simterm]

$ ssh-add -l
2048 SHA256:pTyrGtk1hnNHB6b8ilp5jRe1+K4KrLHg50yUGilApLY Testing key (RSA)

[/simterm]

Deleting key(s)

Use -d to delete one key:

[simterm]

$ ssh-add -d .ssh/test-key
Identity removed: .ssh/test-key (Testing key)

[/simterm]

And -D to delete all keys at once:

[simterm]

$ ssh-add -D
All identities removed.

[/simterm]

Automatically adding keys to `ssh-agent`

To make ssh (and git for example) adding used keys to an ssh-agent without the necessity to run ssh-add manually you can add the AddKeysToAgent parameter to theв ~/.ssh/config and specify one of the following options – yes, confirm or ask (см. SSH_ASKPASS):

[simterm]

$ head -1 .ssh/config
AddKeysToAgent yes

[/simterm]

Let’s check – there is nothing added at this moment:

[simterm]

$ ssh-add -l
The agent has no identities.

[/simterm]

Make a connection, enter a key’s password:

[simterm]

$ ssh -i .ssh/test-key [email protected]
Enter passphrase for key '.ssh/test-key':
...
setevoy@rtfm-do-production:~$

[/simterm]

Disconnect, and check keys in the agent now:

[simterm]

setevoy@rtfm-do-production:~$ logout
Connection to rtfm.co.ua closed.
$ ssh-add -l
2048 SHA256:pTyrGtk1hnNHB6b8ilp5jRe1+K4KrLHg50yUGilApLY Testing key (RSA)

[/simterm]

On the next connection – the ssh client will use the key from the agent and will not ask you for the key’s password again:

[simterm]

$ ssh -i .ssh/test-key [email protected]
...
setevoy@rtfm-do-production:~$

[/simterm]

Running `ssh-agent` with multitype terminals

Another big question is what to do when you have few bash-sessions, for example in various terminals’ tabs, as it will not has the $SSH_AUTH_SOCK variable set and an ssh client will not be able to communicate with an already running ssh-agent instance.

I.e. when you’ll run ssh-add in a new terminal – you’ll see the already mentioned “Could not open a connection to your authentication agent” error:

[simterm]

$ ssh-add -l
Could not open a connection to your authentication agent.

[/simterm]

`~/.bashrc`

There is a few ways to make the initialization of the variables during new bash session initialization, for example, you can add the following to your ~/.bashrc:

if [ -z "$SSH_AUTH_SOCK" ] ; then
  eval `ssh-agent -s`
  ssh-add /home/setevoy/.ssh/test-key
fi

But in this case, each bash-sessions will has its own ssh-agent running, which is not a problem but maybe not what you’d like to have.

Another way could be the following code added to the ~/.bashrc:

ssh-add -l &>/dev/null
if [ "$?" == 2 ]; then
  test -r ~/.ssh-agent-env && \
    eval "$(<~/.ssh-agent-env)" >/dev/null

  ssh-add -l &>/dev/null
  if [ "$?" == 2 ]; then
    (umask 066; ssh-agent > ~/.ssh-agent-env)
    eval "$(<~/.ssh-agent-env)" >/dev/null
    ssh-add /home/setevoy/.ssh/test-key
  fi
fi

Here (see response codes in the ssh-agent documentation):

try to execute ssh-add -l, and redirect output to the /dev/null
check returned code of the previous command:
1. if it is == 2 (error connect to an agent):
  1. check if ~/.ssh-agent-env is present and available for reading, read it and pass its output to the bash
  2. retry ssh-add -l
  3. if code still 2:
    1. create the ~/.ssh-agent-env file with the 660 permissions (read-write for an owner only)
    2. start ssh-agent and redirects its output into the .ssh-agent-env file
    3. read the .ssh-agent-env content and pass it via a pipe to the bash
    4. run ssh-add /home/setevoy/.ssh/test-key

Not a bad solution, and in this way all our sessions will use the same agent, although some guides suggesting to have different agents for personal and work usage

`systemd`

Another solution could be to create a dedicated systemd service by adding a unit file and by running ssh-agent as a systemd service, see the Arch Wiki for the details.

Create a directory if not added yet:

[simterm]

$ mkdir -p .config/systemd/user/

[/simterm]

And create a ~/.config/systemd/user/ssh-agent.service file there:

[Unit]
Description=SSH key agent

[Service]
Type=simple
Environment=SSH_AUTH_SOCK=%t/ssh-agent.socket
ExecStart=/usr/bin/ssh-agent -D -a $SSH_AUTH_SOCK

[Install]
WantedBy=default.target

Next, Wiki told about the ~/.pam_environment file for variables, but in my current case I have Openbox and usually set variables via .config/openbox/autostart file:

[simterm]

$ head -2 .config/openbox/autostart
# ssh-agent.service
SSH_AUTH_SOCK="${XDG_RUNTIME_DIR}/ssh-agent.socket"

[/simterm]

By the way, recalled about about such a thing as setting default values in BASH – BASH: переменные — передача значений по-умолчанию ${var:-defaultvalue}, замена значений — ${var:+alternatevalue} и сообщений — ${var:?message} (Rus)

Now, stop all agents running:

[simterm]

$ killall ssh-agent

[/simterm]

Check the $XDG_RUNTIME_DIR variable value:

[simterm]

$ echo $XDG_RUNTIME_DIR
/run/user/1000

[/simterm]

For now, set the $SSH_AUTH_SOCK variable manually:

[simterm]

$ SSH_AUTH_SOCK="${XDG_RUNTIME_DIR}/ssh-agent.socket"

[/simterm]

And run an agent via systemctl --user:

[simterm]

$ systemctl --user start ssh-agent

[/simterm]

Check it:

[simterm]

$ systemctl --user status ssh-agent
● ssh-agent.service - SSH key agent
  Loaded: loaded (/home/setevoy/.config/systemd/user/ssh-agent.service; disabled; vendor preset: enabled)
  Active: active (running) since Sun 2019-12-01 09:15:18 EET; 2s ago 
Main PID: 497687 (ssh-agent) 
  CGroup: /user.slice/user-1000.slice/[email protected]/ssh-agent.service 
          └─497687 /usr/bin/ssh-agent -D -a /run/user/1000/ssh-agent.socket 

Dec 01 09:15:18 setevoy-arch-pc systemd[670]: Started SSH key agent. 
Dec 01 09:15:19 setevoy-arch-pc ssh-agent[497687]: SSH_AUTH_SOCK=/run/user/1000/ssh-agent.socket; export SSH_AUTH_SOCK; 
Dec 01 09:15:19 setevoy-arch-pc ssh-agent[497687]: echo Agent pid 497687;

[/simterm]

A socket’s variable:

[simterm]

$ echo $SSH_AUTH_SOCK
/run/user/1000/ssh-agent.socket

[/simterm]

And try ssh-add:

[simterm]

$ ssh-add -l
The agent has no identities.

[/simterm]

“It works!” (c)

You can add to autostart now:

[simterm]

$ systemctl --user enable ssh-agent
Created symlink /home/setevoy/.config/systemd/user/default.target.wants/ssh-agent.service → /home/setevoy/.config/systemd/user/ssh-agent.service.

[/simterm]

`~/.xinitrc`

One more way you can use is by adding the agent’s start to the~/.xinitrc.

In this case, when you’ll execute the startx (for example, as in my case, when I have no any login manager, and X.Org is started manually by entering the startx in the console) – at first, an agent will be started and the – an Openbox session, see the documentation:

[simterm]

$ cat ~/.xinitrc
eval $(ssh-agent) &
exec openbox-session

[/simterm]

Also, as already mentioned at the very beginning of this post, there other implementations for the key’s backends that can be used alongside or instead of the ssh-agent – kind of “wrappers” that will be or “proxy” requests from an ssh client to an ssh-agent‘s instance, or will fully replace the ssh-agent itself and will store keys and passwords themselves, but we will speak about them in a following post(s?)..

ssh-agent

Running the agent

Examples

A key generation

Checking SSH-key’s password

ssh-copy-id – copy a key to a remote host

ssh-add

Could not open a connection to your authentication agent

Adding a key

Checking keys

Deleting key(s)

Automatically adding keys to ssh-agent

Running ssh-agent with multitype terminals

~/.bashrc

systemd

~/.xinitrc

`ssh-agent`

`ssh-copy-id` – copy a key to a remote host

`ssh-add`

Automatically adding keys to `ssh-agent`

Running `ssh-agent` with multitype terminals

`~/.bashrc`

`systemd`

`~/.xinitrc`