OpenSSH/Cookbook/Tunnels

 

In tunneling, or port forwarding, a local port is connected to a port on a remote host or vice versa. So connections to the port on one machine are in effect connections to a port on the other machine.

The ssh(1) options -f (go to background), -N (do not execute a remote program) and -T (disable pseudo-tty allocation) can be useful for connections that are used only for creation of tunnels.

Tunneling

Basic SSH Forwarding

In regular port forwarding, connections to a local port are forwarded to a port on a remote machine. This is a way of securing an insecure protocol or of making a remote service appear as local. Here we forwarded VNC in two steps. First make the tunnel:

$ ssh -L 5901:localhost:5901 -l fred desktop.example.org

In that way connections on the local machine made to the forwarded port will in effect be connecting to the remote machine.

Multiple tunnels can be specified at the same time. The tunnels can be of any kind, not just regular forwarding. See the next section below for reverse tunnels. For dynamic forwarding see the section Proxies and Jump Hosts.

$ ssh -L 5901:localhost:5901 \
      -L 5432:localhost:5432 \
      -l fred desktop.example.org

If a connection is only used to create a tunnel, then it can be told not to execute any remote programs (-N), making it a non-interactive session, and also to drop to the background (-f).

$ ssh -fN -L 3128:localhost:3128 -l fred server.example.org

Note that -N will work even if the authorized_keys forces a program using the command="..." option. So a connection using -N will stay open instead of running a program and then exiting.

The three connections above could be saved in the SSH client's configuration file, ~/.ssh/config and even given shortcuts.

Host desktop desktop.example.org
        HostName desktop.example.org
        User fred
        LocalForward 5901 localhost:5901

Host postgres
        HostName desktop.example.org
        User fred
        LocalForward 5901 localhost:5901
        LocalForward 5432 localhost:5432

Host server server.example.org
        HostName server.example.org
        User fred
        ExitOnForwardFailure no
        LocalForward 3128 localhost:3128

Host *
        ExitOnForwardFailure yes

With those settings, the tunnels listed are added automatically when connecting to desktop, desktop.example.org, postgres, server, or server.example.org. The catchall configuration at the end applies to any of the above hosts which have not already set ExitOnForwardFailure to 'no' and the client will refuse to connect if a tunnel cannot be made. The first obtained value for any given configuration directive will be used, but the file's contents can be overidden with run-time options passed on the command line.

Tunneling Via A Single Intermediate Host

Tunneling can go via one intermediate host to reach a second host, and the latter does not need to be on a publicly accessible network. However, the target port on the second remote machine does have to be accessible on the same network as the first. Here, 192.168.0.101 and bastion.example.org must be on the same network and, in addition, bastion.example.org has to be directly accessible to the client machine running ssh(1). So, port 80 on 192.168.0.101 has to be available to the machine bastion.example.org.

$ ssh -fN -L 1880:192.168.0.101:80 -l fred bastion.example.org

Thus, once the tunnel is made, to connect to port 80 on 192.168.2.101 via the host bastion.example.org, connect to port 1880 on localhost. This way works for one or two hosts. It is also possible to chain multiple hosts, using different methods.

Securing a Hop, Tunneling Via One Or More Intermediate Hosts

Here, the idea is to limit the ability of a group of users to the bare minimum needed to pass through a jump host yet still be able to forward ports onward to other machines. If the account is sufficiently locked down then the bastion can only be used for forwarding and not shell access, scripts, or even SFTP. The following settings on the bastion host in sshd_config(5) prevent either shell access or SFTP but still allow port forwarding.

Match Group tunnelers
        ForceCommand /bin/false
        PasswordAuthentication no
        ChrootDirectory %h
        PermitTTY no
        X11Forwarding no
        AllowTcpForwarding yes
        PermitTunnel no
        Banner none

Note that their home directories, but not the files within them, must be owned by root and writable by only root because of the ChrootDirectory configuration directive there in sshd_config(5). Also, because of the PasswordAuthentication configuration directive keys will have to be set up in the home directory in ~/.ssh/authorized_keys, if an alternate location is not already configured.

$ ssh -N -L 9980:localhost:80 -J fred@bastion.example.org fred@192.168.79.124

In that way port 9980 on the client is directed via bastion.example.org through to port 80 on 192.168.79.124.

In cases where the bastion must have a reverse tunnel from the inner host in order to reach it, then the same method works but with the prerequisite of a reverse tunnel from the inner host to the bastion first.

For more about passing through intermediate computers, see the Cookbook section on Proxies and Jump Hosts.

Finding The Process ID (PID) Of A Tunnel Which Is In The Background

When a tunneled connection is sent to the background execution using the -f option for the client, there is not currently an automatic way to find the process ID (PID) of the task sent to the background. Background processes are often used for port forwarding or reverse port forwarding. Here is an example of port forwarding, also called tunneling. The connection is made and then the client goes away, leaving the tunnel available in the background, connecting port 2194 on the local host to port 194 of the remote system's local host.

$ ssh -Nf -L 2194:localhost:194 fred@203.0.113.214

The special $! variable remains empty, even if $? reports success or failure of the action. The reason is that the shell's job control did not put the client into the background. Instead the client runs in the foreground for a moment and then exits normally, after leaving a different process to run in the background via a fork. The process ID of the original client vanishes since that client is gone.

Finding the process ID usually takes at least two steps. Some of the ways to retroactively identify the process involve trying ps(1) and rummaging through the output for all that account's processes, but that is unnecessary effort. If the background SSH client is the most recent one, then pgrep(1) can be used, or else the output needs to be comma delimited and fed into ps(1) via xargs(1) or process substitution.

$ ps uw | less

$ pgrep -n -x ssh

$ pgrep -d, -x ssh | xargs ps -p

$ ps -p $(pgrep -d, -x ssh)

Some variations on the above might be needed depending on operating system if the -d option is not supported.

$ pgrep -x ssh | xargs -n 1 ps -o user,pid,ppid,args -p | awk 'NR==1 || $3==1'
USER       PID  PPID COMMAND
fred     97778     1 ssh -fN -L 8008:localhost:80 fred@203.0.113.8
fred     14026     1 ssh -fN -L 8183:localhost:80 fred@203.0.113.183
fred     79522     1 ssh -fN -L 8228:localhost:80 fred@203.0.113.228
fred     49773     1 ssh -fN -L 8205:localhost:80 203.0.113.205

Either way, note that all the connections running in the background have a Parent Process ID (PPID) of 1, the process control initialization system for the operating system.

Being aware of this shortcoming, a proactive approach can be used with the ControlMaster and ControlPath configuration directives in order to leave a socket to read to get the background task's process ID.

$ ssh -M -S ~/.ssh/sockets/pid.%C -fN -L 5901:localhost:5901 fred@203.0.113.122

$ ssh -S ~/.ssh/sockets/pid.%C -O check 203.0.113.122

The -M option causes the client to go into Master mode for multiplexing using the socket designated by the -S option. Then, after -f forks the client into the background, The control command check will then use the socket to check that the master process is running and report the Process ID.

It is a good idea for the socket to be in an isolated directory not readable or writable by other accounts. Aside from the complexity, a noticeable down side is that it can be possible for the socket to be reused for additional connections. See the Cookbook section on Multiplexing for more about the risks and additional uses.

Reverse Tunneling

Basic SSH Reverse Forwarding

A reverse tunnel forwards connections in the opposite direction of a regular tunnel, that is to say the opposite direction from that which the SSH session is initiated. With remote forwarding as it is also called, an SSH session begins from the local host to a remote host while a port is forwarded from the remote host to one the local host. There are two stages in using reverse tunneling: first connect from endpoint A to endpoint B using SSH with remote forwarding enabled; second connect other systems to the designated port on endpoint B and that port is then forwarded to endpoint A. So while system A initiated the SSH connection to system B, the connections to the designated port on B are sent over to A over the reverse tunnel. Once the SSH connection is made, the reverse tunnel can be used on endpoint B the same as a regular tunnel, even though the endpoint A initiated the SSH connection.

Remote forwarding is method which can be used to forward SSH over SSH in order to work on an otherwise inaccessible system, such as an always-on SBC behind a home router. First, open an SSH session from the inaccessible system to another which is accessible including a designated reverse tunnel. In this example, while the SSH connection is from a local system (endpoint A) to a remote system (endpoint B), that connection contains a reverse tunnel from port 2022 on that remote system (endpoint B) to port 22 over on the local system:

$ ssh -fN -R 2022:localhost:22 -l fred server.example.org

Lastly, using the example above, a connection is made on the remote machine, server.example.org, to the reverse tunnel connection on port 2022. Thus even though the connection is made to port 2022 on locahost on server.example.org, the packets end up over on port 22 on the system which initiated the SSH initial connection carrying the reverse tunnel.

$ ssh -p 2022 -l fred localhost

Thus that example allows SSH access to an otherwise inaccessible system via SSH. SSH goes out, makes a reverse tunnel, then the second system can connect at will to the first via SSH as long as the tunnel persists. If keys and a loop are used to generate the SSH connection with the remote forwarding then the reverse tunnel can be maintained automatically.

The next example makes VNC available over SSH from an otherwise inaccessible system via a second system, server.example.org. Starting from the system with a running VNC server, reverse forward the port for the first VNC display locally to the third VNC display over on server.example.org:

$ ssh -fNT -R 5903:localhost:5901 -l fred server.example.org

Then on the system server.example.org, people can connect to that system's localhost address on its third VNC display and be patched through to the originating system:

$ xvncviewer :3

That also is an example of how the forwarded ports don't have to be the same.

Remote forwarding can be included in the SSH client's configuration file using the RemoteForward directive. See the next subsection for that.

A common use-case for reverse tunneling is when you have to access a system or service which is behind either NAT or a firewall or both and thus incoming SSH connections are blocked, but you have direct access to a second system outside the firewall which can accept incoming connections. In such cases it is easy to make a reverse tunnel from the internal system behind the firewall to the second system on the outside. Once the SSH connection has made the reverse tunnel, to connect to the internal system from outside, other systems can connect to the forwarded port on the remote system. The remote system on the outside then acts as a relay server to forward connections to the initiating system on the inside.

Configuring Remote Forwarding Using The Client Configuration File

The RemoteForward client configuration directive can be used in ssh_config(5) to establish a reverse tunnel from another system. The ForkAfterAuthentication would be -f and SessionType would be -N as run time arguments and, of course, optional in this example which is the same as the first example in the subsection above:

Host server server.example.org
        User fred
        HostName server.example.org
        ForkAfterAuthentication yes
        SessionType none
        RemoteForward 2022 localhost:22

With that in place over on the other system, one can establish a reverse tunnel from server by simply connecting to it by entering ssh server. Then over on server.example.org, connecting to localhost on port 2022 will carry through back to the original system on port 22.

Host server server.example.org
        User fred
        HostName server.example.org
        ForkAfterAuthentication yes
        SessionType none
        RequestTTY no
        RemoteForward 5903 localhost:5901

Likewise that duplicates the second example from the subsection above but in the client configuration file instead of using run time arguments. More about that is covered in the chapter on using The Client Configuration File.

Adding or Removing Tunnels within an Established Connection

It is possible to add or remove tunnels, reverse tunnels, and SOCKS proxies to or from an existing connection using escape sequences. The default escape character is the tilde (~) and the full range of options is described in the manual page for ssh(1). Escape sequences only work if they are the first characters entered on a line and followed by a return. When adding or removing a tunnel to or from an existing connection, ~C, the command line is used.

To add a tunnel in an active SSH session, use the escape sequence to open a command line in SSH and then enter the parameters for the tunnel:

~C
L 2022:localhost:22

To remove a tunnel from an active SSH session is almost the same. Instead of -L, -R, or -D we have -KL, -KR, and -KD plus the port number. Use the escape sequence to open a command line in SSH and then enter the parameters for removing the tunnel.

~C
KL2022

Adding or Removing Tunnels within a Multiplexed Connection

There is an additional option for forwarding when multiplexing. More than one SSH connection can be multiplexed over a single TCP connection. Control commands can be passed to the master process to add or drop port forwarding to the master process.

First a master connection is made and a socket path assigned.

$ ssh -S '/home/fred/.ssh/%h:%p' -M server.example.org

Then using the socket path, it is possible to add port forwarding.

$ ssh -O forward -L 2022:localhost:22 -S '/home/fred/.ssh/%h:%p' fred@server.example.org

Since OpenSSH 6.0 it is possible to cancel specific port forwarding using a control command.

$ ssh -S  "/home/fred/.ssh/%h:%p" -O cancel -L 2022:localhost:22 fred@server.example.org

For more about multiplexing, see the Cookbook section on Multiplexing.

Restricting Tunnels to Specific Ports

By default, port forwarding will allow forwarding to any port if it is allowed at all. The way to restrict which ports a user can use in forwarding is to apply the PermitOpen option on the server side either in the server's configuration or inside the user's public key in authorized_keys. For example, with this setting in sshd_config(5) all users can forward only to port 7654 on the server, if they try forwarding:

PermitOpen localhost:7654

Multiple ports may be specified on the same line if separated by whitespace.

PermitOpen localhost:7654 localhost:3306

If the client tries to forward to a disallowed port, there will be a warning message that includes the text "open failed: administratively prohibited: open failed" while the connection otherwise continues as normal. However, even if the client has ExitOnForwardFailure in its configuration a connection will still succeed, despite the warning message.

However, if shell access is available, it is possible to run other port forwarders, so without further restrictions, PermitOpen is more of a reminder or recommendation than a restriction. But for many cases, such a reminder might be enough.

For reverse tunnels, the PermitListen option is available instead. It determines which port on the remote system is available. So the following, for example, allows ssh -R 2022:localhost:xxxx, where xxxx can be any available port number at the origin of the reverse tunnel, but only 2022 on the far end of the tunnel.

PermitListen localhost:2022

The PermitOpen or PermitListen options can be used as part of one or more Match blocks if forwarding options need to vary depending on various combinations of user, group, client address or network, server address or network, and/or the listening port used by sshd(8). If using the Match criteria to selectively apply rules for port forwarding, it is also possible to prevent the account from getting an interactive shell by setting PermitTTY to no. That will prevent the allocation of a pseudo-terminal (PTY) on the server thus prevent shell access, but allow other programs to still be run unless an appropriate forced command is specified in addition.

Match Group mariadbusers
        PermitOpen localhost:3306
        PermitTTY no
        ForceCommand /usr/bin/true

With that stanza in sshd_config(5) it is only possible to connect by adding the -N option to avoid executing a remote command.

$ ssh -N -L 3306:localhost:3306 server.example.org

The -N option can be used alone or with the -f option which drops the client to the background once the connection is established.

Without the ForceCommand option in a Match block in the server configuration, if an attempt is made to get a PTY by a client that is blocked from getting one, the warning "PTY allocation request failed on channel n" will show up on the client, with n being the channel number, but otherwise the connection will succeed without a remote shell and the port(s) will still be forwarded. Various programs, including shells, can still be specified by the client, they just won't get a PTY. So to really prevent access to the system other than forwarding, a forced command is needed. The tool true(1) comes in handy for that. Note that true(1) might be in a different location on different systems.

Limiting Port Forwarding Requests Using Keys Only

The following authorized_keys line shows the PermitOpen option prepended to a key in order to limit a user connecting with that particular key to forwarding to just port 8765:

permitopen="localhost:8765" ssh-ed25519 AAAAC3NzaC1lZDI1NT...

Multiple PermitOpen options may be applied to the same public key if they are separated by commas and thus a key can allow multiple ports.

By default, shell access is allowed. With shell access it is still possible to run other port forwarders. The no-pty option inside the key can facilitate making a key that only allows forwarding and not an interactive shell if combined with an appropriate forced command using the command option. Here is an example of such a key as it would be listed in authorized_keys:

no-pty,permitopen="localhost:9876",command="/usr/bin/true" ssh-ed25519 AAAAC3NzaC1lZDI1NT...

The no-pty option blocks interactive shell. The client will still connect to the remote server and will allow forwarding but will respond with an error including the message "PTY allocation request failed on channel n". But as mentioned in the previous subsection, there are still a lot of ways around that and adding the command option hinders them.

This method is awkward to lock down. If the account has write access in any way, directly or indirectly, to the authorized_keys file, then it is possible for the user to add a new key or overwrite the existing key(s) with more permissive options. In order to prevent that, the server has to be configured to look for the file in a location outside of the user's reach. See the section on Public Key Authentication for details on that. The methods described above in the previous subsection using sshd_config(5) might be more practical in many cases.