OpenSSH/Cookbook/Public Key Authentication

 

Authentication keys can improve efficiency, if done properly. As a bonus advantage, the passphrase and private key never leave the client. Key-based authentication is generally recommended for outward facing systems so that password authentication can be turned off.

Key-based authentication

OpenSSH can use public key cryptography for authentication. In public key cryptography, encryption and decryption are asymmetric. The keys are used in pairs, a public key to encrypt and a private key to decrypt. The ssh-keygen(1) utility can make RSA, Ed25519, ECDSA, Ed25519-SK, or ECDSA-SK keys for authenticating. Even though DSA keys can still be made, being exactly 1024 bits in size, they are no longer recommended and should be avoided. RSA keys are allowed to vary from 1024 bits on up. The default is now 3072. However, there is only limited benefit after 2048 bits and that makes elliptic curve algorithms preferable. ECDSA can be 256, 384 or 521 bits in size. Ed25519, Ed25519-SK, and ECDSA-SK keys each have a fixed length of 256 bits. Shorter keys are faster, but less secure. Longer keys are much slower to work with but provide better protection, up to a point.

Keys can be named to help remember what they are for. Because the key files can be named anything it is possible to have many keys each named for different services or tasks. The comment field at the end of the public key can also be useful in helping to keep the keys sorted, if you have many of them or use them infrequently.

The process of key-based authentication uses these keys to make a couple of exchanges using the keys to encrypt and decrypt some short message. At the start, a copy of the client's public key is stored on the server and the client's private key is on the client, both stay where they are. The private key never leaves the client. As the client first contacts the server, the server responds by using the client's public key to encrypt a random number and return that encrypted random number as a challenge to the client. The client responds to the challenge by using the matching private key to decrypt the message and extract the random number. The client then makes an MD5 hash of the session ID along with the random number from the challenge and returns that hash to the server. The server then makes its own hash of the session ID and the random number and compares that to the hash returned by the client. If there is a match, the login is allowed. If there is not a match, then the next of any public keys on the server registered as belonging to the same account is tried until either a match is found or all the keys have been tried or the maximum number of failures has been reached.

When an agent is used on the client side to manage authentication, the process is similar. The difference is that ssh(1) passes the challenge off to the agent which then calculates the response and passes it back to ssh(1) which then passes the agent's response back to the server.

Basics of Public Key Authentication

A matching pair of keys is needed for public key authentication and ssh-keygen(1) is used to make the key pair. Out of that pair the public key must be properly stored on the remote host. On the server the public key is added to the designated authorized_keys file for that remote user account. The private key stays stored safely on the client. Once the keys have been prepared they can be used again and again.

There are four steps in preparation for key-based authentication:

1) Prepare the directories where the keys will stay. If either the authorized_keys file or .ssh directory do not exist on the remote machine or the .ssh directory on the client machine, create them and set the permissions correctly. On the client only a directory is needed, but it should not be writable by any account except its owner:

$ mkdir ~/.ssh/
$ chmod 0700 ~/.ssh/

On the remote machine, the .ssh directory is needed as is a special file to store the public keys, the default is authorized_keys.

$ mkdir ~/.ssh/
$ touch ~/.ssh/authorized_keys
$ chmod 0700 ~/.ssh/
$ chmod 0600 ~/.ssh/authorized_keys

2) Create a key pair. The example here creates an Ed25519 key pair in the directory ~/.ssh. The option -t assigns the key type and the option -f assigns the key file a name. It is good to give keys files descriptive names, especially if larger numbers of keys are managed. Below, the public key will be named mykey_ed25519.pub and the private key will be called mykey_ed25519. Be sure to enter a sound passphrase to encrypt the private key using 128-bit AES.

$ ssh-keygen -t ed25519 -f ~/.ssh/mykey_ed25519

Ed25519, Ed25519-SK, and ECDSA-SK keys have a fixed length. For RSA and ECDSA keys, the -b option sets the number of bits used.

Since 6.5 a new private key format is available using a bcrypt(3) key derivative function (KDF) to better protect keys at rest. This new format is always used for Ed25519 keys, and sometime in the future will be the default for all keys. But for right now it may be requested when generating or saving existing keys of other types via the -o option in ssh-keygen(1).

$ ssh-keygen -o -b 4096 -t rsa -f ~/.ssh/mykey_rsa

Details of the new format are found in the source code in the file PROTOCOL.key.

3) Get the keys to the right places. Transfer only the public key to remote machine.

$ ssh-copy-id -i ~/.ssh/mykey_ed25519 fred@remotehost.example.org

If ssh-copy-id(1) is not available, any editor that does not wrap long lines can be used. For example, nano(1) can be started with the -w option to prevent wrapping of long lines. Or another way to set that permanently is by editing nanorc(5). However the authorized_keys file is edited to add the key, the key itself must be in the file whole and unbroken on a single line.

Then if they are not already on the client, transfer both the public and private keys there. It is usually best to keep both the public and private keys together in the directory ~/.ssh/, though the public key is not needed on the client after this step and can be regenerated if it is ever needed again.

4) Test the keys

While still logged in, use the client to start another SSH session in a new window and try authenticating to the remote machine from the client using the private key.

$ ssh -i ~/.ssh/mykey_ed25519 -l fred remotehost.example.org

The option -i tells ssh(1) which private key to try. Close the original SSH session only after verifying that the key-based authentication works.

Once key-based authentication has been verified to be working, it is possible to make permanent shortcuts on the client using ssh_config(5), explained further below. In particular, see the IdentityFile, IdentitiesOnly, and AddKeysToAgent configuration directives, to name three.

➥ Troubleshooting of Key-based Authentication:

If the server refuses to accept the key and fails over to the next authentication method (eg: "Server refused our key"), then there are several possible mistakes to look for on the server side.

One of the most common errors is that the file and directory permissions are wrong. The authorized keys file must be owned by the user in question and not be group writable. Nor may the key file's directory be group or world writable.

$ chmod u=rwx,g=rx,o= ~/.ssh
$ chmod u=rw,g=,o=  ~/.ssh/authorized_keys

Another mistake that can happen is if the key inside the authorized_keys file on the remote host is broken by line breaks or has other whitespace in the middle. That can be fixed by joining up the lines and removing the spaces or by recopying the key more carefully.

And, though it should go without saying, the halves of the key pair need to match. The public key on the server needs to match the private key held on the client. If the public key is lost, then a new one can be generated with the -y option, but not the other way around. If the private key is lost, then the public key should be erased as it is no longer of any use. If many keys are in use for an account, it might be a good idea to add comments to them. On the client, it can be a good idea to know which server the key is for, either through the file name itself or through the comment field. A comment can be added using the -C option.

$ ssh-keygen -t ed25519 -f ~/.ssh/mykey_ed25519 -C "web server mirror"

On the server, it can be important to annotate which client they key is from if there is more than one public key there in an account. There the comment can be added to the authorized keys file on the server in the last column if a comment does not already exist. Again, the format of the authorized keys file is given in the manual page for sshd(8) in the section "AUTHORIZED_KEYS FILE FORMAT". If the keys are not labeled they can be hard to match, which might or might not be what you want.

Associating Keys Permanently with a Server

A key can be specified at run time, but to save retyping the same paths again and again, the Host directive in ssh_config(5) can apply specific settings to a target host. In this case, by changing ~/.ssh/config it is possible to assign particular keys to be tried automatically whenever making a connection to that specific host. After adding the following lines to ~/.ssh/config, all that's needed is to type ssh web1 to connect with the key for that server.

Host web1
	Hostname 198.51.100.32
	IdentitiesOnly yes
	IdentityFile /home/fred/.ssh/web_key_ed25519

The ~/.ssh/config below uses different keys for server versus server.example.org, regardless whether they resolve to the same machine. This is possible because the host name argument given to ssh(1) is not converted to a canonicalized host name before matching.

Host server
	IdentitiesOnly yes
	IdentityFile /home/fred/.ssh/key_a_rsa

Host server.example.org
	IdentitiesOnly yes
	IdentityFile /home/fred/.ssh/key_b_rsa

In this example the shorter name is tried first, but of course less ambiguous shortcuts can be made instead. The configuration file gets parsed on a first-match basis. So the most specific rules go at the beginning and the most general rules go at the end.

Encrypted Home Directories

When using encrypted home directories the keys must be stored in an unencrypted directory. That means somewhere outside the actual home directory which means sshd(8) needs to be configured appropriately to find the keys in that special location.

Here is one method for solving the access problem. Each user is given a subdirectory under /etc/ssh/keys/ which they can then use for storing their authorized_keys file. This is set in the server's configuration file /etc/ssh/sshd_config

AuthorizedKeysFile      /etc/ssh/keys/%u/authorized_keys

Setting a special location for the keys opens up more possibilities as to how the keys can be managed and multiple key file locations can be specified if they are separated by whitespace. The user does not have to have write permissions for the authorized_keys file. Only read permission is needed to be able to log in. But if the user is allowed to add, remove, or change their keys, then they will need write access to the file to do that.

One symptom of having an encrypted home directory is that key-based authentication only works when you are already logged into the same account, but fails when trying to make the first connection and log in for the first time.

Sometimes it is also necessary to add a script or call a program from /etc/ssh/sshrc immediately after authentication to decrypt the home directory.

Passwordless Login

One way of allowing passwordless logins is to follow the steps above, but simply do not enter a passphrase when asked for one while creating the key. Note that using keys that lack a passphrase is very risky, so the key files should be very well protected and kept track of. That includes that they only be used as single-purpose keys as described below. Timely key rotation becomes especially important. In general, it is not a good idea to make a key without a passphrase. A better solution is to have a passphrase and work with an authentication agent in conjunction with a single-purpose key. Most desktop environments launch an SSH agent automatically these days. It will be visible in the SSH_AUTH_SOCK environment variable if it is. On accounts with an agent, ssh-add(1) can load private keys into an available agent.

$ ssh-add ~/.ssh/mykey_ed25519

Thereafter, the client will automatically check the agent for the key when appropriate. If there are many keys in the agent, it will become necessary to set IdentitiesOnly. See the above section on using ~/.ssh/config for that. See [OpenSSH/Cookbook/Public_Key_Authentication#Key-based_Authentication_Using_an_Agent Key-based Authentication Using an Agent] below.

Requiring Both Keys and a Password

While users should have strong passphrases for their keys, there is no way to enforce or verify that. Indeed, since neither the private key nor its the passphrase ever leave the client machine there is nothing that the server can do to have any influence over that. Instead, it is possible to require both a key and a password. Starting with OpenSSH 6.2, it is possible for the server to require multiple authentication methods for login using the AuthenticationMethods directive.

AuthenticationMethods publickey,password

This example from http://man.openbsd.org/sshd_config.5 sshd_config(5)] requires that users first authenticate using a key and it only queries for a password if the key succeeds. Thus with that configuration it is not possible to get to the system password prompt without first authenticating with a valid key. Changing the order of the arguments changes the order of the authentication methods.

Requiring Two or More Keys

Since OpenSSH 6.8, the server now remembers which public keys have been used for authentication and refuses to accept previously-used keys. This allows a set up requiring that users authenticate using two different public keys, maybe one in the file system and the other in a hardware token.

AuthenticationMethods publickey,publickey

The AuthenticationMethods directive, whether for keys or passwords, can also be set on the server under a Match directive to apply only to certain groups or situations.

Requiring Certain Key Types For Authentication

Also since OpenSSH 6.8, the PubkeyAcceptedKeyTypes directive, later changed to PubkeyAcceptedAlgorithms, can specify which key algorithms are accepted for authentication. Those not in the comma-separated pattern list are not allowed.

PubkeyAcceptedAlgorithms ssh-ed25519*,ssh-rsa*,ecdsa-sha2*,sk-ssh-ed25519*,sk-ecdsa-sha2*

Either the actual key types or a pattern can be in the list. Spaces are not allowed in the pattern list. The exact list of key types supported for authentication can be found by the -Q option using the client. The following two lines are equivalent.

$ ssh -Q key-sig | sort

$ ssh -Q PubkeyAcceptedAlgorithms | sort

For host-based authentication, it is the HostbasedAcceptedAlgorithms directive which determines the key types which are allowed for authentication.

Key-based Authentication Using an Agent

When an authentication agent, such as ssh-agent(1), is going to be used, it should generally be started at the beginning of a session and used to launch the login session or X-session so that the environment variables pointing to the agent and its UNIX-domain socket are passed to each subsequent shell and process. Many desktop distros do this automatically upon login or startup.

Starting an agent entails setting a pair of environment variables:

• SSH_AGENT_PID : the process id of the agent
• SSH_AUTH_SOCK : the filename and full path to the UNIX-domain socket

The various SSH and SFTP clients find these variables automatically and use them to contact the agent and try when authentication is needed. However, it is mainly SSH_AUTH_SOCK which is ever used. If the shell or desktop session was launched using ssh-agent(1), then these variables are already set and available. If they are not available, then it is necessary to either set the variables manually inside each shell or for each application in order to use the agent or else to point to the agent's socket using the directive IdentityAgent in the client's configuration file.

Once an agent is available, a relevant private key needs to be loaded before the agent can be used. Once in the agent the private key can then be used many times. Private keys are loaded into an agent with ssh-add(1).

$ ssh-add /home/fred/.ssh/mykey_ed25519
Identity added: /home/fred/.ssh/mykey_ed25519 (/home/fred/.ssh/mykey_ed25519)

Keys stay in the agent for as long as it is running unless specified otherwise. A timeout can be set either with the -t option when starting the agent itself or when actually loading the key using ssh-add(1). In either case, the -t option will set a timeout interval, after which the key will be purged from the agent.

$ ssh-add -t 1h30m /home/fred/.ssh/mykey_ed25519
Identity added: /home/fred/.ssh/mykey_ed25519 (/home/fred/.ssh/mykey_ed25519)
Lifetime set to 5400 seconds

The option -l will list the fingerprints of all of the identities in the agent.

$ ssh-add -l                                    
256 SHA256:77mfUupj364g1WQ+O8NM1ELj0G1QRx/pHtvzvDvDlOk mykey for task x (ED25519)
3072 SHA256:7unq90B/XjrRbucm/fqTOJu0I1vPygVkN9FgzsJdXbk myotherkey rsa for task y (RSA)

It is also possible to remove individual identities from the agent using -d which will remove them one at a time if identified by file name, but only if the file name is given and without the file name of the private key to be remove, -d will fail silently. Using -D instead will remove all of them at once without needing to specify any by name.

By default ssh-add(1) uses the agent connected via the socket named in the environment variable SSH_AUTH_SOCK, if it is set. Currently, that is its only option. However, for ssh(1) an alternative to using the environment variable is the client configuration directive IdentityAgent which tells the SSH clients which socket to use to communicate with the agent. If both the environment variable and the configuration directive are available at the same time, then the value in IdentityAgent takes precedence over what's in the environment variable. IdentityAgent can also be set to none to prevent the connection from trying to use any agent at all.

The client configuration directive AddKeysToAgent can also be useful in getting keys into an agent as needed. When set, it automatically loads a key into a running agent the first time the key is called for if it is not already loaded. Likewise the IdentitiesOnly directive can ensure that the relevant key is offered on the first try. Rather than typing these out whenever the client is run, they can be added to ~/.ssh/config and thereby added automatically for designated host connections.

Agent Forwarding

Agent forwarding is one means of passing through one or more intermediate hosts. However, the -J option for ProxyJump would be a safer option. See Passing Through a Gateway or Two in the section on jump hosts about that. With agent forwarding, intermediate machines forward challenges and responses back and forth between the client and the final destination. This comes with some risks but eliminates the need for using passwords or holding keys on any of these intermediate machines.

A main advantage of agent forwarding is that the private key itself is not needed on any remote machine, thus hindering unwanted file system access to it. Another advantage is that the actual agent to which the user has authenticated does not go anywhere and is thus less susceptible to analysis.

One risk with agents is that they can be re-used to tailgate in if the permissions allow it. Keys cannot be copied this way, but authentication is possible when there are incorrect permissions. Note that disabling agent forwarding does not improve security unless users are also denied shell access, as they can always install their own forwarders.

The risks of agent forwarding can be mitigated by confirming each use of a key by adding the -c option when adding the key to the agent. This requires the SSH_ASKPASS variable be set and available to the agent process, but will generate a prompt on the host running the agent upon each use of the key by a remote system. So if passing through one or more intermediate hosts, it is usually better to instead have the SSH client use stdio forwarding with -W or -J.

On the client side agent forwarding is disabled by default and so if it is to be used it must be enabled explicitly. Put the following line in ssh_config(5) to enable agent forwarding for a particular server:

        Host gateway.example.org
        ForwardAgent yes

On the server side the default configuration files allow authentication agent forwarding, so to use it, nothing needs to be done there, just on the client side. However, again, it would be preferable to take a look at ProxyJump instead.

Old Style, Somewhat Safer SSH Agent Forwarding

The best way to pass through one or more intermediate hosts is to use the ProxyJump option instead of authentication agent forwarding and thereby not risk exposing any private keys. If authentication agent forwarding must be used, then it would be advisable in the interest of following the principle of least privilege to forward an agent containing the minimum necessary number of keys. There are several ways to solve that.

In version 8.8 and earlier a partial solution is to make a one-off, ephemeral agent to hold just the one key or keys needed for the task at hand. Another partial solution would be to set up a user-accessible service at the operating system level and then use ssh_config for the rest.

Automatically launching an ephemeral agent unique to each session can be done by crafting either a special shell alias or function to launch a single-use agent. Either the function or the alias can be written to require confirmation for each requested signature. The following example is an alias is based on an updated blog post by Vincent Bernat on SSH agent forwarding:

$ alias assh="ssh-agent ssh -o AddKeysToAgent=confirm -o ForwardAgent=yes"

Note the use of ssh-agent(1). When invoking that alias, the SSH client will be launched with a unique, ephemeral supporting key agent. The alias sets up a new agent, including setting the two environment variables, and then sets two client options while calling the client. This arrangement still checks with ssh_config(5) for other options and settings. When the SSH session is finished the agent which launched it ends and goes away, thus cleaning up after itself automatically.

Another way is to rely on the client's configuration file for some of the settings. Such methods rely mostly on ssh_config(5) but still require an independent method to launch an ephemeral agent because the OpenSSH client is already running by the time it reads the configuration file and is thus not affected by any changes to environment variables caused by the configuration file and it is through the environment variables that contain information about the agent. However, when the path to the UNIX-domain socket used to communicate with the authentication agent is decided in advance then the IdentityAgent option can point to it once the one-off agent is actually launched. The following uses a specific agent's pre-defined socket whenever connecting to either of two particular domains:

Host *.wikimedia.org *.wmflabs.org
        User fred
        IdentitiesOnly yes
        IdentityFile %d/.ssh/id_cloud_01
        IdentityAgent /run/user/%i/ssh-cloud-01.socket
        ForwardAgent yes
        AddKeysToAgent yes

The %d stands for the path to the home directory and the %i stands for the user id (UID) for the current account. In some cases the %i token might also come in handy when setting the IdentityAgent option inside the configuration file. Again, be careful when forwarding agents with which keys are in the forwarded agent. See the section "TOKENS" in ssh_config(5) for more such abbreviations.

With those configuration settings, the authentication agent must already be up and running and point to the designated socket prior to starting the SSH client for that configuration to work. Additionally, it should place the socket in a directory which is inaccessible to any other accounts. ssh-agent(1) must use the -a option to name the socket:

$ ssh-agent -a /run/user/${UID}/ssh-cloud-01.socket

That agent configuration can be launched manually or via a script or service manager. However, in the interests of privacy and security in general, agent forwarding is to be avoided. The configuration directive ProxyJump is the best alternative and, on older systems, host traversal using ProxyCommand with netcat are preferable. Again, see the section on Proxies and Jump Hosts for how those methods are used.

New Style SSH Agent Destination Constraints

From 8.9 onward, ssh-agent(1) will allow the agent to limit which hosts they will use for authentication as specified by ssh-add(1) using the -h option. These constraints have been added through two agent protocol extensions and a modification to the public key authentication protocol. This feature may evolve, but for now the result is such that keys for account authentication can be loaded into the agent in four ways:

• no limits on forwarding (not recommended)
• local use only, these will not get forwarded
• forwarding, but only to specific remote hosts
• forwarding to specific remote hosts via specified routes

The intent is that the restrictions fail safely so that they do not allow authentication when one or more hosts in the route lack the needed protocol features. The destinations and routes cannot be modified once the keys are loaded, but multiple routes to the same destination can be loaded and the routes can be any number of hops. If the routes need changing, then the key must be reloaded into the agent with the new route or routes.

The general default for the client is to keep keys in the agent for local use only. However, that can be enforced explicitly by adding the -a option when starting the client or else setting the ForwardAgent directive in ssh_config(5) to 'no' in the relevant configuration block.

In order to load keys for unlimited forwarding, which is not the best idea, just add them using ssh-add(1) as normal. Then use the -A option with the client or set the ForwardAgent directive in ssh_config(5) to 'yes' in the relevant configuration block.

In order to limit keys for connection only to a specific remote host, or to load keys for connection to a specific remote host with forwarding via one or more intermediate hosts, use he -h option when loading keys into the agent. Here the one key may be used only to connect to the specific destination:

$ ssh-agent -h server.example.org server.key.ed25519

If an intermediate system is passed through, the best way is to use ProxyJump which is the -J option for the SSH Client. If agent forwarding must be allowed then the tightest way is to constrain which systems may use the keys, again using the -h option.

$ ssh-agent -h middle.example.org -h "middle.example.org>server.example.org" server.key.ed25519

Multiple steps can be included, even multiple routes. They just have to be enumerated explicitly, though patterns may still be used for the destination hosts as well as specific names. Each host in the chain must support these protocol extensions for the connection to complete.

Any keys designated for forwarding are unusable for authentication on any other hosts than those which have been explicitly identified for forwarding. These permitted hosts are identified by host key or host certificate from the known_hosts file or another file designated by the -H option when loading the key with ssh-add(1). If -H is not used at the time the keys are loaded into the agent, then the default known hosts file(s) will be used: ~/.ssh/known_hosts, /etc/ssh/ssh_known_hosts, ~/.ssh/known_hosts2, and /etc/ssh/ssh_known_hosts2.

In the case of keys, the known_hosts list must be maintained conscientiously , perhaps with the help of the UpdateHostkeys and CanonicalizeHostname client configuration directives. Use of certificates requires the agent to only need to be aware of the Certificate Authority (CA).

Again, see Passing Through a Gateway or Two in the section on jump hosts about a way to pass through one or more intermediate machines without needing to forward an SSH agent.

Checking the Agent for Specific Keys

The ssh_add(1) utility's -T option can test whether a specific private key is available in the agent or not by looking up the matching public key. That can be useful in a shell script.

#!/bin/sh

key=/home/fred/.ssh/some.key.ed25519.pub

if ssh-add -T ${key}; then
        echo "Key ${key} Found"
else
        echo "Key ${key} missing"
fi

Or it could be done with an alternate syntax just as well either in a script or in an interactive shell sessions,

$ key=/home/fred/.ssh/some.key.ed25519.pub
$ ssh-add -T ${key} && echo "Key found" || echo "Key missing"

However, if the desired result would be to add key to the agent then the AddKeysToAgent client configuration option can ensure that a specific key is added to the SSH agent upon first use during any given login session. That can be done using -o AddKeysToAgent=yes as a run-time argument, or by modifying ssh_config(5) as appropriate:

Host www
        HostName www.example.com
        IdentityFile %d/.ssh/www.ed25519
        IdentitiesOnly yes
        AddKeysToAgent yes

With those options in the configuration file, the first time ssh www is run the specified key will get added to the agent and remain available.

Key-based Authentication Using A Hardware Security Token

While stand-alone keys have been around for a long time, it has been possible since version 8.2 to use keys backed by hardware security tokens, such as OnlyKey, Yubikey, or many others, though the FIDO2 protocol. The Universal 2nd Factor (U2F) authentication is supported directly in OpenSSH through FIDO2 and does not need third party software. At the moment there are two types of hardware backed keys, ECDSA-SK and Ed25519-SK, but only the latest hardware tokens support the latter. If the key Ed25519-SK format is not supported by the token's firmware, then the following error message will be presented when attempts to use that key type are made:

Key enrollment failed: invalid format

If supported, either key type can be created with ssh-keygen(1). The steps are almost identical to creating normal keys but the token must be available to the system (plugged in) first. Then if called for, the token's PIN must be entered and the token touched or otherwise activated. After that, the key creation proceeds as normal. Mind the key type as specified by the -t option.

$ ssh-keygen -t ed25519-sk -f /home/fred/.ssh/server.ed25519-sk -C "web server for fred"
Generating public/private ed25519-sk key pair.
You may need to touch your authenticator to authorize key generation.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/fred/.ssh/server.ed25519-sk
Your public key has been saved in /home/fred/.ssh/server.ed25519-sk.pub
The key fingerprint is:
SHA256:41wVVDnKJ9gKr2Sj4CFuYMhcNvYebZ6zq0PWyP4rRDo web server
The key's randomart image is:
+[ED25519-SK 256]-+
|           .o... |
|             .o  |
|           +.. . |
|   = .  . ..= .  |
|+ + * + So.. o   |
|o+.EoO *+oo      |
|.o oBo+++o       |
|  o .=.+.        |
| .   .===        |
+----[SHA256]-----+

Once created, the public and private key files get handled like with any other type of key. But when authenticating, the hardware token must be present and activated when called for.

$ ssh -i /home/fred/.ssh/server.ed25519-sk server.example.org
Enter passphrase for key '/home/fred/.ssh/server.ed25519-sk': 
Confirm user presence for key ED25519-SK SHA256:41wVVDnKJ9gKr2Sj4CFuYMhcNvYebZ6zq0PWyP4rRDo

The resulting private key file is not actually the key itself but instead a "key handle" which is used by the hardware security token to derive the real private key on demand at the time it is actually used. As a result, the hardware-backed private key file is useless without the accompanying hardware token. This also means that these key files are not portable across hardware tokens, say when having multiple tokens in reserve or as backup, even when used by the same account. So when multiple hardware tokens are in use, different key pairs must be generated for each token.

Hardware Security Token Resident Private Key

It is possible to store the private key within the token itself, but for the moment it cannot be used directly from inside the token and must first be saved as a file. Also, the key can only be loaded into the FIDO authenticator at the time of creation using the -O resident option with ssh-keygen(1). Otherwise, the process is the same as above.

$ ssh-keygen -O resident -t ed25519-sk -f /home/fred/.ssh/server.ed25519-sk -C "web server for fred"
. . .

When needed, the resident key can be extracted from the FIDO2 hardware token and saved into a file using the -K option. At this stage a passphrase can be added to the file, but no passphrase is kept within the token itself, only an optional PIN protects the key there.

$ ssh-keygen -K
Enter PIN for authenticator: 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Saved ED25519-SK key to id_ed25519_sk_rk

$ mv -i id_ed25519_sk_rk /home/fred/.ssh/server.ed25519-sk

Since the output file name is fixed, any pre-existing file with that name can get overwritten but there will be a warning first. However, it is not recommended to keep the key on the hardware token because it provides more protection when kept separately.

Single-purpose Keys

Tailored single-purpose keys can eliminate use of remote root logins for many administrative activities. A finely tailored sudoers is needed along with an unprivileged account. When done right, it gives just enough access to get the job done, following the security principle of Least Privilege.

Single-purpose keys are accompanied by use of either the ForceCommand directive in sshd_config(5) or the command="..." directive inside the authorized_keys file. The method is to generate a new key pair, transfer the public key to authorized-keys on the remote system, and then prepend the appropriate command or script there to the line with the key.

$ grep '^command' ~/.ssh/authorized_keys
command="/usr/local/bin/somescript.sh" ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEcgdzDvSebOEjuegEx4W1I/aA7MM3owHfMr9yg2WH8H

The command="..." directive inserted there overrides everything else and ensures that when logging in with just that key only the script /usr/local/bin/somescript.sh is run. If it is necessary to pass parameters to the script, have a look at the contents of the SSH_ORIGINAL_COMMAND environment variable and use it in a case statement. Do not ever trust the contents of that variable nor use the contents directly, always indirectly.

Single-purpose keys are useful for allowing only a tunnel and nothing more. The following key will only echo some text and then exit, unless used non-interactively with the -N option.

$ grep '^command' ~/.ssh/authorized_keys
command="/bin/echo do-not-send-commands" ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBzTIWCaILN3tHx5WW+PMVDc7DfPM9xYNY61JgFmBGrA

No matter what the user tries while logging in with that key, the session will only echo the given text and then exits. Using the -N option disables running the remote program, allowing the connection to stay open, allowing a tunnel.

$ ssh -L 3306:localhost:3306 \
	-i ~/.ssh/tunnel_ed25519 \
	-N \
	-l fred \
	server.example.com

That creates a tunnel and stays connected despite a key configuration which would close an interactive session. See also the -n or -f option for ssh(1).

Single-purpose Keys to Avoid Remote Root Access

The easy way is to write a short shell script, place it /usr/local/bin/, and then configure sudoers to allow the otherwise unprivileged account to run just that script and only that script.

%wheel  ALL=(root:root) NOPASSWD: /usr/sbin/service httpd stop
%wheel  ALL=(root:root) NOPASSWD: /usr/sbin/service httpd start

Then the key calls the script using command="..." inside authorized_keys. Here the one key starts the web server, the other stops the web server.

$ grep '^command' ~/.ssh/authorized_keys
command="/usr/bin/sudo /usr/sbin/service httpd stop" ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEcgdzDvSebOEjuegEx4W1I/aA7MM3owHfMr9yg2WH8H
command="/usr/bin/sudo /usr/sbin/service httpd start" ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMidyqZ6OCvbWqA8Zn+FjhpYE6NoWSxVjFnFUk6MrNZ4

Complicated programs like rsync(1), tar(1), mysqldump(1), and so on require an advanced approach when building a single-purpose key. For them, the -v option can show exactly what is being passed to the server so that sudoers can be set up correctly. That way they can be restricted to only access designated parts of the file system. For example, here is what ssh -v shows from one particular usage of rsync(1), note the "Sending command" line:

$ rsync -e 'ssh -v' fred@server.example.org:/etc/ ./backup/etc/
. . .
debug1: Sending command: rsync --server --sender -e.LsfxC . /etc/
. . .

That output can then be added to sudoers so that the key can do only that function.

%backup ALL=(root:root) NOPASSWD: /usr/bin/rsync --server --sender -e.LsfxC . /etc/

Then to tie it all together, the account "backup" needs a key:

$ grep '^command' ~/.ssh/authorized_keys
command="/usr/bin/rsync --server --sender -e.LsfxC . /etc/" ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMm0rs4eY8djqBb3dIEgbQ8lmdlxb9IAEuX/qFCTxFgb

Many of these programs have a ---dry-run or equivalent option. Remember to use it when figuring out the right settings.

Read-only Access to Keys

In some cases it is necessary to prevent accounts from being able to changing their own authentication keys. However, such situations may be a better case for using certificates. However, if done with keys it is accomplished by putting the key file in an external directory where the user has read-only access, both to the directory and to the key file. Then the AuthorizedKeysFile directive assigns where sshd(8) looks for the keys and can point to a secured location for the keys instead of the default location.

A good alternate location could be a new directory /etc/ssh/authorized_keys which could store the selected accounts' key files there. The change can be made to apply to only a group of accounts by putting the settings under a Match directive. The default location for keys on most systems is usually ~/.ssh/authorized_keys.

Match Group sftpusers
	AuthorizedKeysFile /etc/ssh/authorized_keys/%u

Then the permissions there would allow the keys to be read but not written:

$ ls -dhln /etc/ssh/
drwxr-x--x 3 0 0 4.0K Mar 30 22:16 /etc/ssh/authorized_keys/

$ ls -dhln /etc/ssh/*.pub
-rw-r--r-- 1 0 0 173 Mar 23 13:34 /etc/ssh/fred
-rw-r--r-- 1 0 0  93 Mar 23 13:34 /etc/ssh/user1
-rw-r--r-- 1 0 0 565 Mar 23 13:34 /etc/ssh/user2
. . .

The keys could even be in within subdirectories, though the same restrictions apply regarding permissions and ownership.

For chrooted SFTP, the method is the same to keep the key files out of reach of the accounts:

Match Group sftpusers
	ChrootDirectory /home
	ForceCommand internal-sftp -d %u
	AuthorizedKeysFile /etc/ssh/authorized_keys/%u

Of course a Match directive is not essential. The settings could be made to apply to all accounts by putting the directive in the main part of the server configuration file instead.

Mark Public Keys as Revoked

Keys can be revoked. Keys that have been revoked can be stored in /etc/ssh/revoked_keys, a file specified in sshd_config(5) using the directive RevokedKeys, so that sshd(8) will prevent attempts to log in with them. No warning or error on the client side will be given if a revoked key is tried. Authentication will simply progress to the next key or method.

The revoked keys file should contain a list of public keys, one per line, that have been revoked and can no longer be used to connect to the server. The key cannot contain any extras, such as login options or it will be ignored. If one of the revoked keys is tried during a login attempt, the server will simply ignore it and move on to the next authentication method. An entry will be made in the logs of the attempt, including the key's fingerprint. See the section on logging for a little more on that.

RevokedKeys /etc/ssh/revoked_keys

The RevokedKeys configuration directive is not set in sshd_config(5) by default. It must be set explicitly if it is to be used. This is another situation that might be better fulfilled through using certificate since a validity interval can be set in any combination of seconds, minutes, hours, days, or weeks can be set for certificates while keys are valid indefinitely.

Key Revocation Lists

A Key Revocation List (KRL) is a compact, binary form of representing revoked keys and certificates. In order to use a KRL, the server's configuration file must point to a valid list using the RevokedKeys directive. KRLs themselves are generated with ssh-keygen(1) and can be created from scratch or edited in place. Here a new one is made, populated with a single public key:

$ ssh-keygen -kf  /etc/ssh/revoked_keys -z 1 ~/.ssh/old_key_rsa.pub

Here an existing KRL is updated by adding the -u option:

$ ssh-keygen -ukf /etc/ssh/revoked_keys -z 2 ~/.ssh/old_key_dsa.pub

Once a KRL is in place, it is possible to test if a specific key or certificate is in the revocation list.

$ ssh-keygen -Qf  /etc/ssh/revoked_keys ~/.ssh/old_key_rsa.pub

Only public keys and certificates will be loaded into the KRL. Corrupt or broken keys will not be loaded and will produce an error message if tried. Like with the regular RevokedKeys list, the public key destined for the KRL cannot contain any extras like login options or it will produce an error when an attempt is made to load it into the KRL or search the KRL for it.

Verify a Host Key by Fingerprint

The above examples have been about using keys to authenticate the client to the server. A different context in which keys are used is when the server identifies itself to the client, which happens automatically at the beginning of each non-multiplexed session. In order for that identification to happen the client acquires a public key from the server, usually on or prior to first contact, which it can subsequently use to ensure that it is connecting to the same server again and not an impostor. The default locations for storing these acquired host keys on the client are in /etc/ssh/ssh_known_hosts, if managed by the system administrator, or in ~/.ssh/known_hosts if managed by the client's own account. The format of the contents is a line with a host address and its matching public key. The file is described in detail in the sshd(8) manual page in the section "SSH_KNOWN_HOSTS FILE FORMAT".

When connecting for the first time to a remote host, the server's host key should be verified in order to ensure that the client is connecting to the right machine and not an impostor or anything else. Usually this verification is done by comparing the fingerprint of the server's host key rather than trying to compare the whole key itself. By default the client will show the fingerprint if the key is not already found in the known_hosts register.

$ ssh -l fred server.example.org
The authenticity of host 'server.example.org (192.0.32.10)' can't be established.
ECDSA key fingerprint is SHA256:LPFiMYrrCYQVsVUPzjOHv+ZjyxCHlVYJMBVFerVCP7k.
Are you sure you want to continue connecting (yes/no)?

That can be compared to a fingerprint received out of band, say by post, e-mail, SMS, courier, and so on. Specifically, the example represents the key's fingerprint as a base64 encoded SHA256 checksum. That is the default style. The fingerprint can also be displayed as an MD5 hash in hexadecimal instead by passing the client's FingerprintHash configuration directive as a runtime argument or setting it in ssh_config(5).

$ ssh -o FingerprintHash=md5 host.example.org
The authenticity of host 'host.example.org (192.0.32.203)' can't be established.
RSA key fingerprint is MD5:10:4a:ec:d2:f1:38:f7:ea:0a:a0:0f:17:57:ea:a6:16.
Are you sure you want to continue connecting (yes/no)?

But the default in new versions is SHA256 in base64 has a lower chance of collision.

In OpenSSH 6.7 and earlier, the client showed fingerprints as a hexadecimal MD5 checksum instead a of the base64-encoded SHA256 checksum currently used:

$ ssh -l fred server.example.org
The authenticity of host 'server.example.org (192.0.32.10)' can't be established. 
RSA key fingerprint is 4a:11:ef:d3:f2:48:f8:ea:1a:a2:0d:17:57:ea:a6:16. 
Are you sure you want to continue connecting (yes/no)?

Another way of comparing keys is to use the ASCII art visual host key. See further below about that.

Downloading keys

Even though a host’s key is usually displayed for review the first time the SSH client tries to connect, it can also be fetched on demand at any time using ssh-keyscan(1):

$ ssh-keyscan host.example.org  
# host.example.org SSH-2.0-OpenSSH_8.2
host.example.org ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLC2PpBnFrbXh2YoK030Y5JdglqCWfozNiSMjsbWQt1QS09TcINqWK1aLOsNLByBE2WBymtLJEppiUVOFFPze+I=
# host.example.org SSH-2.0-OpenSSH_8.2
host.example.org ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9iViojCZkcpdLju7/3+OaxKs/11TAU4SuvIPTvVYvQO32o4KOdw54fQmd8f4qUWU59EUks9VQNdqf1uT1LXZN+3zXU51mCwzMzIsJuEH0nXECtUrlpEOMlhqYh5UVkOvm0pqx1jbBV0QaTyDBOhvZsNmzp2o8ZKRSLCt9kMsEgzJmexM0Ho7v3/zHeHSD7elP7TKOJOATwqi4f6R5nNWaR6v/oNdGDtFYJnQfKUn2pdD30VtOKgUl2Wz9xDNMKrIkiM8Vsg8ly35WEuFQ1xLKjVlWSS6Frl5wLqmU1oIgowwWv+3kJS2/CRlopECy726oBgKzNoYfDOBAAbahSK8R
# host.example.org SSH-2.0-OpenSSH_8.2
host.example.org ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDDOmBOknpyJ61Qnaeq2s+pHOH6rdMn09iREz2A/yO2m

Once a key is acquired, its fingerprint can be shown using ssh-keygen(1). This can be done directly with a pipe.

$ ssh-keyscan host.example.org | ssh-keygen -lf - 
# host.example.org SSH-2.0-OpenSSH_8.2
# host.example.org SSH-2.0-OpenSSH_8.2
# host.example.org SSH-2.0-OpenSSH_8.2
256 SHA256:sxh5i6KjXZd8c34mVTBfWk6/q5cC6BzR6Qxep5nBMVo host.example.org (ECDSA)
3072 SHA256:hlPei3IXhkZmo+GBLamiiIaWbeGZMqeTXg15R42yCC0 host.example.org (RSA)
256 SHA256:ZmS+IoHh31CmQZ4NJjv3z58Pfa0zMaOgxu8yAcpuwuw host.example.org (ED25519)

If there is more than one public key type is available from the server on the port polled, then ssh-keyscan(1) will fetch each of them. If there is more than one key fed via stdin or a file, then ssh-keygen(1) will process them in order. Prior to OpenSSH 7.2 manual fingerprinting was a two step process, the key was read to a file and then processed for its fingerprint.

$ ssh-keyscan -t ed25519 host.example.org > key.pub
# host.example.org SSH-2.0-OpenSSH_6.8
$ ssh-keygen -lf key.pub                       
256 SHA256:ZmS+IoHh31CmQZ4NJjv3z58Pfa0zMaOgxu8yAcpuwuw host.example.org (ED25519)

Note that some output from ssh-keyscan(1) is sent to stderr instead of stdout.

A hash, or fingerprint, can be generated manually with awk(1), sed(1) and xxd(1), on systems where they are found.

$ awk '{print $2}' key.pub | base64 -d | md5sum -b | sed 's/../&:/g; s/: .*$//'
$ awk '{print $2}' key.pub | base64 -d | sha256sum -b | sed 's/ .*$//' | xxd -r -p | base64

It is possible to find all hosts from a file which have new or different keys from those in known_hosts, if the host names are in clear text and not stored as hashes.

$ ssh-keyscan -t rsa,ecdsa -f ssh_hosts | \
  sort -u - ~/.ssh/known_hosts | \
  diff ~/.ssh/known_hosts -

Using ssh-keyscan(1) with ssh_config(5)

The utility ssh-keyscan(1) does not parse ssh_config(5). That is in part to keep the code base simple. There are a lot of configuration options which would be complicated to implement, including but not limited to ProxyJump, ProxyCommand, Match, BindInterface, and CanonicalizeHostname . Resolving host names via the client configuration file can be done by wrapping the utility in a short shell function:

my-ssh-keyscan() {
	for host in "$@" ; do
		ssh-keyscan $(ssh -G "$host" | awk '/^hostname/ {print $2}')
	done
}

That shell function uses the -G option of ssh(1) to resolve each host name using ssh_config(5) and then check the resulting host name for SSH keys.

ASCII Art Visual Host Key

An ASCII art representation of the key can be displayed along with the SHA256 base64 fingerprint:

$ ssh-keygen -lvf key 
256 SHA256:BClQBFAGuz55+tgHM1aazI8FUo8eJiwmMcqg2U3UgWU www.example.org (ED25519)
+--[ED25519 256]--+
|o+=*++Eo         |
|+o .+.o.         |
|B=.oo.  .        |
|*B.=.o .         |
|= B *   S        |
|. .@ .           |
| +..B            |
|  *. o           |
| o.o.            |
+----[SHA256]-----+

In OpenSSH 6.7 and earlier the fingerprint is in MD5 hexadecimal form.

$ ssh-keygen -lvf key 
2048 37:af:05:99:e7:fb:86:6c:98:ee:14:a6:30:06:bc:f0 www.example.net (RSA) 
+--[ RSA 2048]----+ 
|          o      | 
|         o .     | 
|        o o      | 
|       o +       | 
|   .  . S        | 
|    E ..         | 
|  .o.* ..        | 
|  .*=.+o         | 
|  ..==+.         | 
+-----------------+

More on Verifying SSH Keys

Keys on the client or the server can be verified against known good keys by comparing the base64-encoded SHA256 fingerprints.

Verifying Stray Client Keys

Sometimes is is necessary to compare two uncertain key files to check if they are part of the same key pair. However, public keys are more or less disposable. So the easy way in such situations on the client machine is to just rename or erase the old, problematic, public key and replace it with a new one generated from the existing private key.

$ ssh-keygen -y -f ~/.ssh/my_key_rsa

But if the two parts must really be compared, it is done in two steps using ssh-keygen(1). First, a new public key is re-generated from the known private key and used to make a fingerprint to stdout. Next, the fingerprint of the unknown public key is generated for comparison. In this example, the private key my_key_a_rsa and the public key my_key_b_rsa.pub are compared:

$ ssh-keygen -y -f my_key_a_rsa | ssh-keygen -l -f -
$ ssh-keygen -l -f my_key_b_rsa.pub

The result is a base64-encoded SHA256 checksum for each key with the one fingerprint displayed right below the other for easy visual comparison. Older versions don't support reading from stdin so an intermediate file will be needed then. Even older versions will only show an MD5 checksum for each key. Either way, automation with a shell script is simple enough to accomplish but outside the scope of this book.

Verifying Server Keys

Reliable verification of a server's host key must be done when first connecting. It can be necessary to contact the system administrator who can provide it out of band so as to know the fingerprint in advance and have it ready to verify the first connection.

Here is an example of the server's RSA key being read and its fingerprint shown as SHA256 base64:

$ ssh-keygen -lf /etc/ssh/ssh_host_rsa_key.pub     
3072 SHA256:hlPei3IXhkZmo+GBLamiiIaWbeGZMqeTXg15R42yCC0 root@server.example.net (RSA)

And here the corresponding ECDSA key is read, but shown as an MD5 hexadecimal hash:

$ ssh-keygen -E md5 -lf /etc/ssh/ssh_host_ecdsa_key.pub
256 MD5:ed:d2:34:b4:93:fd:0e:eb:08:ee:b3:c4:b3:4f:28:e4 root@server.example.net (ECDSA)

Prior to 6.8, the fingerprint was expressed as an MD5 hexadecimal hash:

$ ssh-keygen -lf /etc/ssh/ssh_host_rsa_key.pub 
2048 MD5:e4:a0:f4:19:46:d7:a4:cc:be:ea:9b:65:a7:62:db:2c root@server.example.net (RSA)

It is also possible to use ssh-keyscan(1) to get keys from an active SSH server. However, the fingerprints still needs to be verified out of band.

Warning: Remote Host Identification Has Changed!

If a server's key does not match what the client finds has been recorded in either the system's or the local account's authorized_keys files, then the client will issue a warning along with the fingerprint of the suspicious key.

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
SHA256:GkoIDP/d0I6KA9IQyOB9iqL+Rzpxx9LhlSJPCEfjVQ4.
Please contact your system administrator.
Add correct host key in /home/fred/.ssh/known_hosts to get rid of this message.
Offending RSA key in /home/fred/.ssh/known_hosts:19
  remove with:
  ssh-keygen -f "/home/fred/.ssh/known_hosts" -R "server.example.com"
RSA host key for server.example.com has changed and you have requested strict checking.
Host key verification failed.

Three reasons for the warning are common.

One reason is that the server's keys were replaced, often because the server's operating system was reinstalled without backing up the old keys. Another reason can be when the system administrator has phased out deprecated or compromised keys. However that can be planned better and if there is time to plan the migration, new keys can just be added to the server and have the clients use the UpdateHostKeys option so that the new keys are accepted if the old keys match. A third situation is when the connection is made to the wrong machine, such as when the remote system changes IP addresses because of dynamic address allocation.

In all three cases where the key has changed there is only one thing to do: contact the system administrator and verify the key. Ask if the OpenSSH-server was recently reinstalled, or was the machine restored from an old backup? Keep in mind that the system administrator may be you yourself in some cases.

The case which is rather rare but serious enough that it should be ruled out for sure is that the wrong machine is part of a man-in-the-middle attack.

In all four cases, an authentic key fingerprint can be acquired by any method where it is possible to verify the integrity and origin of the message, for example via PGP-signed e-mail. If physical access is possible, then use the console to get the right fingerprint. Once the authentic key fingerprint is available, return to the client machine where you got the error and remove the old key from ~/.ssh/known_hosts

$ ssh-keygen -R server.example.org

Then try logging in, but compare the key fingerprints first and proceed if and only if the key fingerprint matches what you received out of band. If the key fingerprint matches, then go through with the login process and the key will be automatically added. If the key fingerprint does not match, stop immediately and figure out what you are connecting to. It would be a good idea to get on the phone, a real phone not a computer phone, to the remote machine's system administrator or the network administrator.

Multiple Keys for a Host, Multiple Hosts for a Key in known_hosts

Multiple host names or IP addresses can use the same key in the known_hosts file by using pattern matching or simply by listing multiple systems for the same key. That can be done in either the global list of keys in /etc/ssh/ssh_known_hosts and the local, account-specific lists of keys in each account's ~/.ssh/known_hosts file. Labs, computational clusters, and similar pools of machines can make use of keys in that way. Here is a key shared by three specific hosts, identified by name:

server1,server2,server3 ssh-rsa AAAAB097y0yiblo97gvl...jhvlhjgluibp7y807t08mmniKjug...==

Or a range can be specified by using globbing to a limited extent in either /etc/ssh/ssh_known_hosts or ~/.ssh/known_hosts.

172.19.40.* ssh-rsa AAAAB097y0yiblo97gvl...jhvlhjgluibp7y807t08mmniKjug...==

Conversely, for multiple keys for the same address, it is necessary to make multiple entries in either /etc/ssh/ssh_known_hosts or ~/.ssh/known_hosts for each key.

server1 ssh-rsa AAAAB097y0yiblo97gvljh...vlhjgluibp7y807t08mmniKjug...==
server1 ssh-rsa AAAAB0liuouibl kuhlhlu...qerf1dcw16twc61c6cw1ryer4t...==
server1 ssh-rsa AAAAB568ijh68uhg63wedx...aq14rdfcvbhu865rfgbvcfrt65...==

Thus in order to get a pool of servers to share a pool of keys, each server-key combination must be added manually to the known_hosts file:

server1 ssh-rsa AAAAB097y0yiblo97gvljh...07t8mmniKjug...==
server1 ssh-rsa AAAAB0liuouibl kuhlhlu...qerfw1ryer4t...==
server1 ssh-rsa AAAAB568ijh68uhg63wedx...aq14rvcfrt65...==

server2 ssh-rsa AAAAB097y0yiblo97gvljh...07t8mmniKjug...==
server2 ssh-rsa AAAAB0liuouibl kuhlhlu...qerfw1ryer4t...==
server2 ssh-rsa AAAAB568ijh68uhg63wedx...aq14rvcfrt65...==

Though upgrading to certificates might be a more appropriate approach that manually updating lots of keys.

Another way of Dealing with Dynamic (roaming) IP Addresses

It is possible to manually point to the right key using HostKeyAlias either as part of ssh_config(5) or as a runtime parameter. Here the key for machine Foobar is used to connect to host 192.168.11.15

$ ssh -o StrictHostKeyChecking=accept-new \
      -o HostKeyAlias=foobar   \
      192.168.11.15

This is useful when DHCP is not configured to try to keep the same addresses for the same machines over time or when using certain stdio forwarding methods to pass through intermediate hosts.

Hostkey Update and Rotation in known_hosts

A protocol extension to rotate weak public keys out of known_hosts has been in OpenSSH from version 6.8 and later. With it the server is able to inform the client of all its host keys and update known_hosts with new ones when at least one trusted key already known. This method still requires the private keys be available to the server so that proofs can be completed. In ssh_config(5), the directive UpdateHostKeys specifies whether the client should accept updates of additional host keys from the server after authentication is completed and add them to known_hosts. A server can offer multiple keys of the same type for a period before removing the deprecated key from those offered, thus allowing an automated option for rotating keys as well as for upgrading from weaker algorithms to stronger ones. See also RFC 4819: Secure Shell Public Key Subsystem about key management standards.