OpenSSH/Cookbook/Certificate-based Authentication

 

Certificates are keys which have been signed by another key. The key used for such signing is called the certificate authority. It is made in advance and set aside, reserved for signing only. Other parties use the signing key's public half to verify the authenticity of the signed key being used for server identification, in the case of a host certificate, or for login, in the case of a user certificate .

In the interest of privilege separation, make separate certificate authorities for host certificates and user certificates if both are going to be used. As of the time of this writing, either of the elliptical curve algorithms are a good choice.

Overview of SSH Certificates

X.509 is an ITU Telecommunications Standardization Sector standard which defines the format of public key certificates.

When using certificates either the client or the server are pre-configured to accept keys which have themselves been signed by another key specially designated and set aside for just the purpose of signing the keys actually used for work. That other key is known as a certificate authority, or CA for short. As with normal keys, certificates are used to generate signatures used in authentication. However, rather than looking up the matching public key in a file, the public key is filed with a signature and the signature is used to verify the public key and then the public key is used to ensure that the negotiations are happening with a client in possession of the matching private key. So a prerequisite for using certificates is at least a passing familiarity with normal SSH. See the chapter on Public Key Authentication. The mechanics of normal key-based authentication are described briefly there in the introduction. SSH uses its own simpler certificate format and not the X.509 certificate format.

Two of the main advantages of certificates over keys are that they can use an expiration date, or even a date range of validity, and that they eliminate need for trust-on-first-use or complicated key verification methods. Mostly they facilitate large scale deployments by easing the processes of key approval and distribution and provide a better option than copying the same host keys across multiple destinations.

User certificates authenticate users to their accounts on the servers. Host certificates authenticate servers to the clients, proving that the clients are connecting to the right system. The use of a principals field to designate users versus hosts is the main difference between host and user certificates. In host certificates, the principals field refers to the server names represented by the certificate. In user certificates that field refers to the accounts which are allowed to use the certificate for logging in. Additional limitations just as specific source addresses and forced commands are available for user certificates. Date and time of validity are possible for both. Host certificates and user certificates should use separate certificate authorities. For a more authoritative resource, see the "CERTIFICATES" section of ssh-keygen(1).

SSH User Certificates

User certificates authenticate the user to a server or other remote device, in other words they allow people and scripts to log in. Authenticating the client to the server by means of user certificates means that the authorized_keys file on the server is no longer needed. Instead the user certificate, really a signed key, is checked against the certificate authority to verify if the signature is valid. If it is, then the login process can proceed. Another advantage is that the signatures can be designated as valid only for a range of dates. So in practice, it is possible to have an expiration date. User certificates are also tied to specific accounts, referred to as 'principals'. The principal(s) allowed to use the signed key must be designated or the server will not accept use of the authentication key even if it is properly signed.

It is even possible to restrict the source of incoming connections or force specific commands using options like force-command and source-address similar to normal SSH public keys. These restrictions are set at the time of signing. If those options are to be changed, the key must be resigned or else become invalid.

Files Associated with SSH User Certificates

There are five files associated with user certificates. The Certificate Authority must be kept safe and stored out of band. If it is lost then no new user accounts can be added to the pool and a new certificate authority must be established. If it falls into the wrong hands then an attacker could use it to pass their accounts off as legitimate users:

• Certificate Authority - a private SSH key generated for signing other keys

The next one is kept on the server itself:

• Certificate Public Key - the public component of the certificate authority

The last three files are kept on the client systems:

• Private SSH Key - the private key used for authenticating to the server or remote device
• Public SSH Key - the matching public key which has been signed by the Certificate Authority
• User Certificate - the signature made for the User Public Key using the Certificate Authority

Optionally, the user certificate and its key can be associated permanently with a remote server or device using the | ssh_config file, either globally or per-account, via the CertificateFile and IdentityFile directives.

Steps for Working with SSH User Certificates

There are four steps in working with user certificates. Step one, creation of a certificate authority, is done just once per pool of users or scripts. Step two, adjusting the servers' configurations, can be repeated for as many server systems or remote devices as are needed. Step three, signing the user keys, is done once per user account. Step four, deploying the user certificate, can be done for as many clients as allowed by the principals and any source-address limitations that might be included.

1. Creating a Certificate Authority

It is a very good idea to keep separate certificate authorities for hosts and users. So, in the interest of privilege separation, make a separate certificate authority for user certificates even if you already have one for host certificates.

$ ssh-keygen -t ed25519 -f ~/.ssh/user_ca_key \
        -C 'User Certificate Authority for *.example.com'

The private key created here should be kept somewhere other than the servers. However, the servers will have access to the public component so as to be able to verify the signature that will be put forth by the clients.

2. Storing the Public Component of the Certificate Authority on the Server

The server needs to be able to look up the matching certificate in order to validate the signature on user keys. That is set in the SSH daemon's configuration. Copy the host certificate to the same place the host keys are stored. Then point the OpenSSH server to the certificate authority's public component using the TrustedUserCAKeys directive in |sshd_config(5)

TrustedUserCAKeys /etc/ssh/user_ca_key.pub

Double check to make sure the file permissions are correct.

$ ls -lhn /etc/ssh/user_ca_key.pub
-rw-r--r--  1 0  0   114B May  4 16:38 /etc/ssh/user_ca_key.pub

Then any number of accounts can be used with that certificate authority.

3. Processing an Existing Public Key

The users must have a key pair already made and then submit the public key component of the pair for signing. Sign it and transfer the signed copies back to the right person. Delete any artifacts of this process immediately, since extra public keys lying around can only cause clutter at best. Here a public key named server01.ed25519.pub has been accepted and a certificate is made with it.

$ ssh-keygen -s user_ca_key -I 'edcba' -z '0002' -n fred \
        server01.ed25519.pub

The resulting certificate will be named server01.ed25519-cert.pub and will have the internal ID "edcba" and an internal serial number "2". Both the ID and the serial number must be calculated externally. For successful logins the certificate's id and serial fields will be included in the log. See the section Logging and Troubleshooting for more depth on the topic. It is a very good idea to list a principal for the certificate, even for user certificates. The principal listed in the certificate does need to match the account it will log in to.

Even though the public key itself is not strictly needed after that on the client side for logging in, it can be good for the client to keep. But if the public key has been lost, a new one can be regenerated from the private key, though not the other way around. When the private key is gone, it is gone. So keep a proper backup schedule. If a file exists with the name the public key should have, it had better be the public key itself or else the login attempt will fail.

The logistics for getting the public key and delivering the certificate are outside the scope of this book. But at this point, the resulting certificate should be transferred back to the person working with the key that was submitted.

4. Logging in Using an SSH User Certificate

On the client side, both the user certificate and the private key it corresponds to are needed to log in.

$ ssh -o CertificateFile=server01.ed25519-cert.pub -i server01.ed25519 \
        fred@server01.example.org

Once things are working manually a shortcut can be made using ssh_config(5) on the client. Use of IdentitiesOnly might also be needed if an agent is used and there are multiple keys in the agent.

Host server01 server01.example.org
        Hostname server01.example.org
        User fred
        IdentitiesOnly yes
        IdentityFile /home/fred/.ssh/server01.ed25519
        CertificateFile /home/fred/.ssh/server01.ed25519-cert.pub

With those settings, running ssh server01 on that client will try to apply both the designated key and its corresponding user certificate and designated principal.

SSH Host Certificates

One of the main uses of keys by OpenSSH is the authentication of a remote host to a client so that the client can verify that it is connecting directly to the right system and not an impostor of via an intruder in between. For that, once acknowledged, the known_hosts file usually keeps a copy of the public key in a register of host keys paired with host names or IP addresses.

The difficulty with host keys in general is in populating the known_hosts file on clients for large pools of client machines or when connecting to new systems for the first time. In a data center or lab or Internet of Things deployment, machines are always coming and going. That means new SSH host keys each time. If there are a lot of hosts involved, then that adds up to a lot of keys in the register. Using a host certificate instead, an arbitrarily large pool of hosts using the same certificate authority need only one entry in the known_hosts register, even as new hosts are added to the pool. By signing the host keys which a new server or device uses to identify itself, it is still possible to roll out new systems with unique keys but have them recognized correctly and safely by clients on the first try without risking the potential for a man-in-the-middle.

By using host certificates, these identifying host keys are signed and the signature can be verified against the agreed upon certificate authority, thus greatly easing the otherwise involved process of collecting and verifying the host's public key when making the first connection.

Files Associated with SSH Host Certificates

There are five files involved in using host certificates. Like with user certificates, the certificate authority must be kept safe. The same precautions apply as for user certificates but for hosts rather than the accounts on them:

• Certificate Authority - a private key generated for signing other keys

The next three files are kept on the server itself.

• Certificate public key - the public component of the certificate authority
• Host Public Key - the actual key that the SSH daemon uses to identify itself to the clients
• Host Certificate - the signature made for the Host Public Key using the Certificate Authority

Then on the clients, either in the client's register or the system-wide register of recognized hosts:

• known_hosts - contains a reference to the host certificate and its principals

When the clients find and use a valid host certificate, no entry for the individual host will be added to the known_hosts register.

Steps for Using SSH Host Certificates

There are four general stages in working with host certificates. Step one, creation of a certificate authority, is done just once per server or pool of servers or devices. Step two, signing the host keys, is done once per server device as is step three. Step four, configuring the clients, can be repeated for as many client machines or individual login accounts as needed.

With each step, mind the paths. There is no one-size-fits-all solution, so it will be necessary to decide where the files should go.

1. Creating a Certificate Authority

Again, a certificate authority, or CA, is just another SSH key. However rather than using it for authenticating the servers or clients directly, it is used to sign and then validate the other keys which are then actually used for authentication.

$ ssh-keygen -t ecdsa -f ~/.ssh/ca_key \
        -C 'Host Certficate Authority for *.example.com'

The private key created here should be kept somewhere other than the servers where they will be used.

The public key in this step must be distributed out of band to the clients which will then use it to verify host identities upon first connection.

2. Fetching and Signing the Host Key

Only the public key gets signed. Fetch the remote host's public host key via a reliable method, sign it, and then upload the resulting certificate to the server. The -h option indicates that this shall be a host certificate and the -s option points to the key used to do the signing. Here a copy of the host key ssh_host_ecdsa_key.pub has been acquired from its server and will be worked on locally:

$ ssh-keygen -h -s ~/.ssh/ca_key -V '+1d' -I abcd -z 00001 \
         -n server.example.com ./ssh_host_ecdsa_key.pub
Enter passphrase: 
Signed host key /etc/ssh/ssh_host_ecdsa_key-cert.pub: id "abcd" serial 1 for server.example.com valid from 2020-05-05T09:51:00 to 2020-05-06T09:52:01

The validity interval set by the -V option is time span relative to the date and time when the key signed. So it would be possible to make a certificate valid for only an hour tomorrow using the formula -V '+1d2h:+1d3h'. If no start time is set, then the value is interpreted as the stopping time. If a specific stopping time or date is required, that is best done by having a script calculate that and then call ssh_key-gen(1). If no time is given at all, the certificate will be considered valid indefinitely.

The -I option assigns a label for the purpose of identifying the certificate.

The -z option manually assigns a serial number to the certificate. That serial number must be extracted from the old certificate and then incremented if it is to be kept in sequence. The default is to not have a serial number. The -n option assigns a set of principals, that would be which hosts may use the certificate in the context of host certificates.

The contents of the certificate can be reviewed using the -L option.

$ ssh-keygen -L -f ssh_host_ecdsa_key-cert.pub                   
ssh_host_ecdsa_key-cert.pub:
        Type: ecdsa-sha2-nistp256-cert-v01@openssh.com host certificate
        Public key: ECDSA-CERT SHA256:kVSFLH5MP/3uJWU57JxD8xVFs7ia8Pww8/ro+pq4S50
        Signing CA: ECDSA SHA256:INewUSvbnfVbgUhtLBhh+XKL0uN99qbXjsi0jvD/IGI (using ecdsa-sha2-nistp256)
        Key ID: "abcd"
        Serial: 1
        Valid: from 2020-05-05T09:51:00 to 2020-05-06T09:52:01
        Principals: 
                server.example.com
        Critical Options: (none)
        Extensions: (none)

The certificate for the public host key must be transferred over to where the server can use it. That usually means the same directory where the regular public host key is found, which is/etc/ssh/ by default. Check to be sure that the certificate has the right permissions after copying it in place.

$ ls /etc/ssh/ssh_host_ecdsa_key*.pub
ls -nlh /etc/ssh/ssh_host_ecdsa_key*.pub
-rw-r--r--  1 0  0   653 May  4 16:49 /etc/ssh/ssh_host_ecdsa_key-cert.pub
-rw-r--r--  1 0  0   172 Feb 21 16:09 /etc/ssh/ssh_host_ecdsa_key.pub

3. Publishing the Host Certificate.

Once the host certificate is in place, the SSH daemon on the remote host must be pointed at the host certificate. See sshd_config(5) for more.

HostCertificate /etc/ssh/ssh_host_ecdsa_key-cert.pub

The SSH daemon must then be instructed to reload its configuration file. The exact method varies from sytem to system but ultimately the daemon will receive a HUP signal.

4. Updating Clients to Acknowledge the Designated Certificate Authority

Finally add a reference to the certificate authority in the client's known_hosts file:

@cert-authority *.example.com ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFJQHK0uoOpBfynyKrjF/SjsLMFewUAihosD6UL3/HkaFPI1n3XAg9D7xePyUWf8thR2e0QVl5TeGLdFiGyCgt0=

That's it. However, it is important to realize that each of the above steps will fail more or less silently and the client will fall back to the usual approach of verifying the host's identity without reporting any error for its part. Any useful debugging information will be in the daemon's logs and even that will be of limited nature. See the chapter on Logging and Troubleshooting for examples.

Orchestration of Certificate Deployment

Automatically deploying certificates, either user certificates or host certificates, is beyond the scope of this book. There are many ways to do it and many factors involved. The details depend not only on which orchestration software is used, but also which specific distro or operating system is in place at the end points. In short, figure out how to do it manually and then figure out how to automate that process using the work flow allowed by the deployment scripts or software.

However, it is important to note that the certificate authority can be kept in an agent. Investigate about the -U option for signing.

Limiting User Certificates

Various limitations can be bound to a user certificate inside the certificate itself. These are mostly specified using the -O option when signing the key and include a validity interval, a forced command, source address ranges, disabling pseudo-terminal allocation, and others. See the manual page for ssh-keygen(1) for an authoritative list. These limitations can be combined or used separately. The examples below will address them separately for clarity.

Time Limitations for User Certificates

The certificate can be made valid for a planned time period, called here as a validity interval. The validity interval can have a specific starting date and time, ending date and time, or both. However, each certificate may have only a single validity interval.

Validity intervals are specified with the -V option and can use an absolute date and time range or a relative one. Using the key example from the user certificate section above, the following would limit when the certificate would be valid. Specifically, it is good for a five minute period on June 24, 2020 from 1:55pm through 2pm.

$ ssh-keygen -V '202006241355:202006241400' \
        -s user_ca_key -I 'edcba' -z '0003' -n fred \
        server01.ed25519.pub

Be sure to look closely at the resulting output to ensure that the range is what it needs to be.

Relative intervals can be used, too. Here is a certificate which is good for only five minutes, starting right away:

$ ssh-keygen -V ':+5m' \
        -s user_ca_key -I 'edcba' -z '0004' -n fred \
        server01.ed25519.pub

Note that the seconds are included and are counted from when the signing is initiated not when the passphrase is eventually entered and the signing finally completed. So if the certificate signing is initiated at 35 seconds past the top of the minute, the expiration time will also be at 35 seconds past the fifth minute. And, again, look closely at the resulting output.

Forced Commands with User Certificates

A user certificate can be tied to a specific command on the server by using the -O option as it is created. In this example, the certificate which only ever show the time and date whenever it is used to connect to the SSH server:

$ ssh-keygen -O force-command='/bin/date +"%T %F"' \
        -s user_ca_key -I 'edcba' -z '0005' -n fred \
        server01.ed25519.pub

If there is a forced command in both the certificate and sshd_config(5), then the latter takes precedence. Any command that was passed as a run time argument is overridden, yet can be found in the SSH_ORIGINAL_COMMAND environment variable. Commands that come from inside the certificate won't affect the SSH_ORIGINAL_COMMAND variable and will have to be parsed from the certificate itself, which will be held in the ephemeral file pointed to by the SSH_USER_AUTH environment variable.

$ awk '/^publickey/ {print $2,$3}' ${SSH_USER_AUTH} \
        | ssh-keygen -Lf -

The file will only exist while the session is still open. The SSH_USER_AUTH variable itself will only be set if the SSH server's configuration has ExposeAuthInfo set to 'yes' and the default is 'no'.

Source Address Restrictions on User Certificates

A certificate can be limited to a specific CIDR range.

$ ssh-keygen -O source-address='198.51.100.0/24,203.0.113.0/26' \
        -s user_ca_key -I 'edcba' -z '0006' -n fred \
        server01.ed25519.pub

The CIDR range must be valid.

Viewing Limitations on User Certificates

If the certificate is at hand, it is possible to look at it in detail and see which limitations apply, certificate-side. Below is an example certificate for the account 'fred', two LAN ranges, and just over one hour of access.

$ ssh-keygen -Lf server01.ed25519-cert.pub

server01.ed25519-cert.pub:
        Type: ssh-ed25519-cert-v01@openssh.com user certificate
        Public key: ED25519-CERT SHA256:hSy7QrAApIU1LgDCUrtBK2F2TZxhvincnJ0djSDow7I
        Signing CA: ED25519 SHA256:dVgTW1INbhvHjHbeAe10R9Niu8BpejifO286RZ7/niU (using ssh-ed25519)
        Key ID: "edcba"
        Serial: 7
        Valid: from 2020-06-24T15:17:00 to 2020-06-24T16:23:47
        Principals: 
                fred
        Critical Options: 
                force-command /bin/date +"%T",source-address=192.168.0.0/16,10.11.9.0/26
        Extensions: 
                permit-X11-forwarding
                permit-agent-forwarding
                permit-port-forwarding
                permit-pty
                permit-user-rc

Obviously the certificate itself cannot show any additional restrictions made server-side in the SSH server's configuration.

If the user certificate is not at hand, but is used for authentication, then limitations and all other embedded characteristics can be gleaned by using the SSH_USER_AUTH variable provided by the ExposeAuthInfo option in sshd_config(5) to fetch the certificate from the server. The certificate itself will leave the SSH_ORIGINAL_COMMAND variable alone, so the temporary file will be the only way to see what was actually in the certificate. Again, the certificate file pointed to by the SSH_USER_AUTH variable will only exist while the session is open.