Hashicorp Vault SSH secret engine with OpenSSH 8.2 and later

OpenSSH login security

If you have ever used OpenSSH with authorized keys, you will know whole process. Also you know, that in order to use it, you must first add your OpenSSH public key to ~/.ssh/authorized_keys file.

It’s very convenient if you have few servers. But what if you have dozens or hundreds? What if you have a lot of consultants, which come and go, and you need to change access immediately? Well, in such case one of solutions would be to use SSH certificate authority, like some companies do. One of SSH CA solutions will be Hashicorp Vault SSH secrets engine.

I will not go to the details on how to setup for Hashicorp Vault SSH secrets engine, official documentation is very good. I assume you already have this setup and have successfully used it in production for some time now. I would like to talk about small caveat which appeared in OpenSSH 8.2 and might block you from using Vault SSH CA.

So lets imagine situation. You want to install few new servers with Ubuntu 20.04 LTS. All seems to be fine: you installed your servers, booted them up, all your services are working, OpenSSH has started. Then you add your tested sshd_config(5) changes, which are working on Ubuntu 18.04 LTS and provide SSH CA functionality. Restart OpenSSH.

And now you try to login via SSH (with signed public SSH key) to freshly installed server and bang, you have Password prompt which you haven’t seen for a while. What‘s going on?

You enable DEBUG2 level or higher on SSH and try login again. And in the logs you might see:

Jun 26 09:43:37 your-server sshd[36287]: debug2: userauth_pubkey: valid user admin querying public key rsa-sha2-512-cert-v01@openssh.com XXXXXXXXXXX
Jun 26 09:43:37 your-server sshd[36287]: userauth_pubkey: certificate signature algorithm ssh-rsa: signature algorithm not supported [preauth]

Strange, it says you have valid SSH signed key, but signature algorithm is not supported. Strange, right? Answer is simple, it appears that OpenSSH doesn’t trust RSA/SHA1 CA signing algorithm anymore.

So you might think, right, let me check if I can change Vault SSH signing mechanism. It might involve a lot of changes, but (for the sake of security) we can do it. But no, you can’t (at least for now). Right, so you have now few options: downgrade OpenSSH, disable SSH CA or allow OpenSSH server trust your CA certificate. There is always a choice.

If you choose to go with Vault SSH secret engine, you will need to add in sshd_config such line to your setup and restart OpenSSH server:

CASignatureAlgorithms ^ssh-rsa

What it does? According to documentation:

CASignatureAlgorithms
 Specifies which algorithms are allowed for signing of certificates by certificate authorities (CAs). The default is:
ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521, ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa

ssh(1) will not accept host certificates signed using algorithms other than those specified.

Note ^ in front of ssh-rsa. As explained in section of PubkeyAcceptedKeyTypes in the same documentation:

If the specified list begins with a ‘^’ character, then the specified key types will be placed at the head of the default set.

After all those change you have again working authentication mechanism, which you enjoyed. Just a small note: keep an eye on Vault issue, maybe they will change CA key signing algorithm in new release of Vault and you will have to re-distribute new CA pub key to all of your servers.