OpenSSH/Cookbook/Multiplexing
Multiplexing is the ability to send more than one signal over a single line or connection. In OpenSSH, multiplexing can re-use an existing outgoing TCP connection for multiple concurrent SSH sessions to a remote SSH server, avoiding the overhead of creating a new TCP connection and reauthenticating each time.
Advantages of Multiplexing
editAn advantage of SSH multiplexing is that the overhead of creating new TCP connections and negotiating the secure connection is eliminated. The overall number of connections that a machine may accept is a finite resource and the limit is more noticeable on some machines than on others and varies greatly depending on both load and usage. There is also a significant delay when opening a new connection. Activities that repeatedly open new connections can be significantly sped up using multiplexing.
The difference between multiplexing and standalone sessions can be seen by comparing the tables below. Both are selected output from netstat -nt slightly edited for clarity. We see in Table 1, "SSH Connections, Separate", that without multiplexing each new login creates a new TCP connection, one per login session. Following that we see in the other table, Table 2, "SSH Connections, Multiplexed", that when multiplexing is used, new logins are channelled over the already established TCP connection.
# Local Address Foreign Address State
# one connection
tcp 0 0 192.168.x.y:45050 192.168.x.z:22 ESTABLISHED
# two separate connections
tcp 0 0 192.168.x.y:45050 192.168.x.z:22 ESTABLISHED
tcp 0 0 192.168.x.y:45051 192.168.x.z:22 ESTABLISHED
# three separate connections
tcp 0 0 192.168.x.y:45050 192.168.x.z:22 ESTABLISHED
tcp 0 0 192.168.x.y:45051 192.168.x.z:22 ESTABLISHED
tcp 0 0 192.168.x.y:45052 192.168.x.z:22 ESTABLISHED
Table 1: SSH Connections, Separate
Both tables show TCP/IP connections associated with SSH sessions. The table above shows a new TCP/IP connection for each new SSH connection. The table below shows a single TCP/IP connection despite multiple active SSH sessions.
# Local Address Foreign Address State
# one connection
tcp 0 0 192.168.x.y:58913 192.168.x.z:22 ESTABLISHED
# two multiplexed connections
tcp 0 0 192.168.x.y:58913 192.168.x.z:22 ESTABLISHED
# three multiplexed connections
tcp 0 0 192.168.x.y:58913 192.168.x.z:22 ESTABLISHED
Table 2: SSH Connections, Multiplexed
As we can see with multiplexing, only a single TCP connection is set up and used regardless of whether or not there are multiple SSH sessions carried over it.
Or we can compare the time it takes to run true(1) on a slow remote server, using time(1). The two commands would be something like time ssh server.example.org true versus time ssh -S ./path/to/somesocket server.example.org true and use keys with an agent for each of them. First without multiplexing, we see the normal connection time:
real 0m0.658s
user 0m0.016s
sys 0m0.008s
Then we do the same thing again, but with a multiplexed connection to see a faster result:
real 0m0.029s
user 0m0.004s
sys 0m0.004s
The difference is quite large and will definitely add up for any activity where connections are made repeatedly in rapid succession. The speed gain for multiplexed connections doesn't come with the master connection, that is normal speed, but with the second and subsequent multiplexed connections. The overhead for a new SSH connection remains, but the overhead of a new TCP connection is avoided. The second and later connections will reuse the established TCP connection over and over and not need to create a new TCP connection for each new SSH connection.
Setting Up Multiplexing
editThe OpenSSH client supports multiplexing its outgoing connections, since version 3.9 (August 18, 2004)[1], using the ControlMaster, ControlPath and ControlPersist configuration directives which get defined in ssh_config(5). The client configuration file usually defaults to the location ~/.ssh/config. All three directives are described in the manual page for ssh_config(5). See also the "TOKENS" section there for the list of tokens available for use in the ControlPath. Any valid tokens used are expanded at run time.
ControlMaster determines whether ssh(1) will listen for control connections and what to do about them. ControlPath sets the location for the control socket used by the multiplexed sessions. These can be either globally or locally in ssh_config(5) or else specified at run time. Control sockets are removed automatically when the master connection has ended. ControlPersist can be used in conjunction with ControlMaster. If ControlPersist is set to 'yes', then it will leave the master connection open in the background to accept new connections until either killed explicitly or closed with -O or ends at a pre-defined timeout. If ControlPersist is set to a time, then it will leave the master connection open for the designated time or until the last multiplexed session is closed, whichever is longer.
Here is a sample excerpt from ssh_config(5) applicable for starting a multiplexed session to machine1.example.org via the shortcut machine1.
Host machine1
HostName machine1.example.org
ControlPath ~/.ssh/controlmasters/%r@%h:%p
ControlMaster auto
ControlPersist 10m
With that configuration, the first connection to machine1 will create a control socket in the directory ~/.ssh/controlmasters/ Then any subsequent connections, up to 10 by default as set by MaxSessions on the SSH server, will re-use that control path automatically as multiplexed sessions. Confirmation of each new connection can be required if ControlMaster set to 'autoask' instead of 'auto'.
Please note with the settings above that the control socket would be placed into the folder ~/.ssh/controlmasters/. If that folder doesn't exist first, the SSH client will exit with a specific error about unix_listener complaining that the file or path does not exist:
unix_listener: cannot bind to path /home/fred/.ssh/controlmasters/fred@machine1.example.org:22.EOWsvm5xFt30O6CB: No such file or directory
The option -O ssh(1) can be used to manage the connection using the same shortcut configuration. To cancel all existing connections, including the master connection, use 'exit' instead of 'stop'.
$ ssh -O check machine1
Master running (pid=14379)
$ ssh -O stop machine1
Stop listening request sent
$ ssh -O check machine1
Control socket connect(/Users/Username/.ssh/sockets/machine1): No such file or directory
In that example, the status of the connection is checked first. Then the master connection is told not to accept further multiplexing requests and finally we check again that no control socket is available.
Manually Establishing Multiplexed Connections
editMultiplexed sessions need a control master to connect to. The run time parameters -M and -S also correspond to ControlMaster and ControlPath, respectively. So first an initial master connection is established using -M when accompanied by the path to the control socket using -S.
$ ssh -M -S /home/fred/.ssh/controlmasters/fred@server.example.org:22 server.example.org
Then subsequent multiplexed connections are made in other terminals. They use ControlPath or -S to point to the control socket.
$ ssh -S /home/fred/.ssh/controlmasters/fred@server.example.org:22 server.example.org
Note that the control socket is named "fred@server.example.org:22", or %r@%h:%p, to try to make the name unique. The combination %r, %h and %p stand for the remote user name, the remote host and the remote host's port. The control sockets should be given unique names.
Multiple -M options place ssh(1) into master mode with confirmation required before slave connections are accepted. This is the same as ControlMaster=ask. Both require X to ask for the confirmation.
Here is the master connection for the host server.example.org as set to ask for confirmation of new multiplexed sessions:
$ ssh -MM -S ~/.ssh/controlmasters/%r@%h:%p server.example.org
And here is a subsequent multiplexed connection:
$ ssh -S ~/.ssh/controlmasters/%r@%h:%p server.example.org
The status of the control master connection can be queried using -O check, which will tell if it is running or not.
$ ssh -O check -S ~/.ssh/controlmasters/%r@%h:%p server.example.org
If the control session has been stopped, the query will return an error about "No such file or directory" even if there are still multiplexed sessions running because the socket is gone.
Alternately, instead of specifying -M and -S as runtime options, configuration options can be spelled out in full using -o for easier transfer to the client configuration file when figured out.
The way below sets configuration options as run time parameters by first starting a control master.
$ ssh -o "ControlMaster=yes" -o "ControlPath=/home/fred/.ssh/controlmasters/%r@%h:%p" server.example.org
Then subsequent sessions are connected to the control master via socket at the end of the control path.
$ ssh -o "ControlPath=/home/fred/.ssh/controlmasters/%r@%h:%p" server.example.org
And of course all that can be put into ssh_config(5) as shown in the previous section. Starting with 6.7, the combination of %r@%h:%p and variations on it can be replaced with %C which by itself generates a SHA1 hash from the concatenation of %l%h%p%r.
$ ssh -S ~/.ssh/controlmasters/%C server.example.org
There are two advantages with that. One is that the hash can be shorter than its combined elements while still uniquely identifying the connection. The other is that it obfuscates the connection information which would have been otherwise displayed in the socket's name.
Ending Multiplexed Connections
editOne way to end multiplexed sessions is to exit all related SSH sessions, including the control master. If the control master has been placed in the background using ControlPersist, then it will be necessary to stop it with -O and either 'stop' or 'exit'. That also requires knowing the full path to and filename of the control socket as used in the creation of the master session, if it has not been defined in a shortcut in ssh_config(5).
$ ssh -O stop server1
$ ssh -O stop -S ~/.ssh/controlmasters/%C server1.example.org
The multiplex command -O stop will gracefully shutdown multiplexing. After issuing the command, the control socket is removed and no new multiplexed sessions are accepted for that master. Existing connections are allowed to continue and the master connection will persist until the last multiplexed connection closes.
In contrast, the multiplex command -O exit removes the control socket and immediately terminates all existing connections.
Again, the directive ControlPersist can also be set to timeout after a set period of disuse. The time interval there is written in a time format listed in sshd_config(5) or else defaults to seconds if no units are given. It will cause the master connection to automatically close if it has no client connections for the specified time.
Host server1
HostName server1.example.org
ControlPath ~/.ssh/controlmasters/%C
ControlMaster yes
ControlPersist 2h
The above example lets the control master timeout after 2 hours of inactivity. Care should be used with persistent control sockets. A user that can read and write to a control socket can establish new connections without further authentication.
Multiplexing Options
editThe values for configuring session multiplexing can be set either in the user-specific ssh_config(5), or the global /etc/ssh/ssh_config, or using parameters when running from the shell or a script. ControlMaster can be overridden in the run time arguments to re-use an existing master when ControlMaster is set to 'yes', by setting it explicitly to 'no':
$ ssh -o "ControlMaster=no" server.example.org
ControlMaster accepts five different values: 'no', 'yes', 'ask', 'auto', and 'autoask'.
- 'no' is the default. New sessions will not try to connect to an established master session, but additional sessions can still multiplex by connecting explicitly to an existing socket.
- 'yes' creates a new master session each time, unless explicitly overridden. The new master session will listen for connections.
- 'ask' creates a new master each time, unless overridden, which listen for connections. If overridden, ssh-askpass(1) will popup an message in X to ask the master session owner to approve or deny the request. If the request is denied, then the session being created falls back to being a regular, standalone session.
- 'auto' creates a master session automatically but if there is a master session already available, subsequent sessions are automatically multiplexed.
- 'autoask' automatically assumes that if a master session exists, that subsequent sessions should be multiplexed, but ask first before adding a session.
Refused connections are logged to the master session.
Host *
ControlMaster ask
ControlPath can be a fixed string or include any of several pre-defined variables described in the TOKENS section of the ssh_config(5). %L is for the first component of the local host name and %l for the full local host name. %h is the target host name and %n is the original target host name and %p is for the destination port on the remote server. %r is for the remote user name and %u for the user running ssh(1). Or they can be combined as %C which is a SHA1 hash produced from %l%h%p%r.
Host *
ControlMaster ask
ControlPath ~/.ssh/controlmasters/%C
ControlPersist accepts 'yes', 'no' or a time interval. If a time interval is given, the default is in seconds. Units can extend the time to minutes, hours, days, weeks or a combination. If 'yes' the master connection stays in the background indefinitely.
Host *
ControlMaster ask
ControlPath ~/.ssh/controlmasters/%C
ControlPersist 10m
Port Forwarding After the Fact
editIt is possible to request port forwarding without having to establish new connections. Here we forward port 8080 on the local host to port 80 on the remote host using -L:
$ ssh -O forward -L 8080:localhost:80 -S ~/.ssh/controlmasters/fred@server.example.org:22 server.example.org
The same can be done for remote forwarding, using -R. The escape sequence ~C is not available for multiplexed sessions, so -O forward is the only way to add port forwarding on the fly.
Port forwarding can be canceled without having to close any sessions using -O cancel.
$ ssh -O cancel -L 8080:localhost:80 -S ~/.ssh/controlmasters/fred@server.example.org:22 server.example.org
The exact same syntax used for forwarding is used for cancelling. However, there is no way currently to look up which ports are being forwarded and how they are forwarded.
Additional Notes About Multiplexing
editNever use any publicly accessible directory for the control path sockets. Place those sockets in a directory somewhere else, one to which only your account has access. For example ~/.ssh/socket/ would be a much safer choice and /tmp/ would be a bad choice.
In cases where it is useful to have a standing connection, it is always possible to combine multiplexing with other options, such as -f or -N to have the control master drop to the background upon connection and not load a shell.
In sshd_config(5), the directive MaxSessions specifies the maximum number of open sessions permitted per network connection. This is used when multiplexing ssh sessions over a single TCP connection. Setting MaxSessions to 1 disables multiplexing and setting it to 0 disables login/shell/subsystem sessions completely. The default is 10. The MaxSessions directive can also be set under a Match conditional block so that different settings are available for different conditions.
Errors Preventing Multiplexing
editThe directory used for holding the control sockets must be on a file system that actually allows the creation of sockets. AFS is one example that doesn't, as are some implementations of HFS+. If an attempt is made at creating a socket on a file system that does not allow it, an error something like the following will occur:
$ ssh -M -S /home/fred/.ssh/mux 192.0.2.57
muxserver_listen: link mux listener /home/fred/.ssh/mux.vjfeIFFzHnhgHoOV => /home/fred/.ssh/mux: Operation not permitted
A similar issue was seen when trying to make Unix domain sockets on OverlayFS filesystems, which are often used with Docker, prior to Linux 4.7 kernel[2].
If the file system cannot be reconfigured to allow sockets, then the only other option is to place the control path socket somewhere else on a file system that does support creation of sockets.
Observing Multiplexing
editIt is possible to make some rough measurements to show the differences between multiplexed connections and one-off connections as shown in the tables and figures above.
Measuring the Number of TCP Connections
editTables 1 and 2 above use output from netstat(8) and awk(1) to show the number of TCP connections corresponding to the number of SSH connections.
#!/bin/sh
netstat -nt | awk 'NR == 2'
ssh -f server.example.org sleep 60
echo # one connection
netstat -nt | awk '$5 ~ /:22$/'
ssh -f server.example.org sleep 60
echo # two connections
netstat -nt | awk '$5 ~ /:22$/'
ssh -f server.example.org sleep 60
echo # three connections
netstat -nt | awk '$5 ~ /:22$/'
echo Table 1
|
And |
#!/bin/sh
netstat -nt | awk 'NR == 2'
ssh -f -M -S ~/.ssh/demo server.example.org sleep 60
echo # one connection
netstat -nt | awk '$5 ~ /:22$/'
ssh -f -S ~/.ssh/demo server.example.org sleep 60 &
echo # two connections
netstat -nt | awk '$5 ~ /:22$/'
ssh -f -S ~/.ssh/demo server.example.org sleep 60 &
echo # three connections
netstat -nt | awk '$5 ~ /:22$/'
echo Table 2
|
The sleep(1) period can be increased if more delay is needed. While the connections are active, it is also possible to look on the server using ps(1), such as with ps uwx, to see the processes as a means of verifying that multiple connections are indeed being made.
Measuring Response Time
editMeasuring the response time requires setting up keys with an agent first so that the agent handles the authentication and eliminates authentication as a source of delay. See the section on keys, if needed. All of the response time tests below will depend on using keys for authentication.
For a one-off connection, just add time(1) to check how long access takes.
$ time ssh -i ~/.ssh/rsakey server.example.org true
For a multiplexed connection, a master control session must be established. Then subsequent connections will show an increase in speed.
$ ssh -f -M -S ~/.ssh/demo -i ~/.ssh/rsakey server.example.org
$ time ssh -S ~/.ssh/demo -i ~/.ssh/rsakey server.example.org true
$ time ssh -S ~/.ssh/demo -i ~/.ssh/rsakey server.example.org true
These response times are approximate but the difference is nonetheless large enough to be seen between one-off connections and multiplexed connections.
Keeping sessions open
editProbably you also want to keep your sessions open when inactive. You can configure it using ServerAliveInterval and ServerAliveCountMax options, see ssh_config(5) for the details.
Multiplexing HTTPS and SSH Using sslh
editA different kind of multiplexing is when more than one protocol is carried over the same port. sslh does just that with SSL and SSH. It figures out what kind of connection is coming in and forwards it to the appropriate service. Thus it allows a server to receive both HTTPS and SSH over the same port, making it possible to connect to the server from behind restrictive firewalls in some cases. It does not hide SSH. Even a quick scan for SSH on the listening port, say with scanssh(1), will show that it is there. Please note that this method only works with simpler packet filters, such as PF or nftables(8) and its front-ends, firewalld and UFW, which filter based on the destination port and won't fool protocol analysis and application layer filters, such as Zorp, which filter based on the actual protocol used. Here's how sslh(8) can be installed in four steps:
- First install your web server and configure it to accept HTTPS requests. Be sure that it listens for HTTPS on the localhost only. It can help in that regard if it is on a non-standard port, say 2443, instead of 443.
- Next, set your SSH server to accept connections on the localhost for port 22. It really could be any port, but port 22 is the standard for SSH.
- Next, create an unprivileged user to run sslh(8). The example below has the user 'sslh' for the unprivileged account.
- Lastly, install and launch sslh(8) so that it listens on port 443 and forwards HTTPS and SSH to the appropriate ports on the local host. Substitute the external IP number for your machine. The actual name of the executable and path may vary from system to system:
$ /usr/local/sbin/sslh-fork -u sslh -p xx.yy.zz.aa:443 --tls 127.0.0.1:2443 --ssh 127.0.0.1:22
Another option is to use a configuration file with sslh(8) and not pass any parameters at runtime. There should be at least one sample configuration file, basic.cfg or example.cfg, included in the package when it is installed. The finished configuration file should look something like this:
user: "sslh";
listen: ( { host: "xx.yy.zz.aa"; port: "443" } );
on-timeout: "ssl";
protocols:
(
{ name: "ssh"; host: "localhost"; port: "22"; probe: "builtin"; },
{ name: "ssl"; host: "localhost"; port: "2443"; probe: "builtin"; }
);
Mind the quotes, commas, and semicolons.
If an old version of SSH is used in conjunction with now-deprecated TCP Wrappers, as described in hosts_access(5), then the option service: works to provide the name of the service that they need.
{ name: "ssh"; service: "ssh"; host: "localhost"; port: "22"; probe: "builtin"; },
If TCP Wrappers are not used, which is most likely the case, then service: is not needed.
Runtime parameters override any configuration file settings that may be in place. sslh(8) supports the protocols HTTP, SSL, SSH, OpenVPN, tinc, and XMPP out of the box. But actually any protocol that can be identified by regular expression pattern matching can be used. There are two variants of sslh, a forked version (sslh or sslh-fork) and a single-threaded version (sslh-select). See the project web site for more details: https://www.rutschle.net/tech/sslh/README.html
References
edit- ↑ "OpenSSH 3.9 Release Notes". OpenSSH. 2004-08-18. Retrieved 2018-12-16.
- ↑ Szeredi, Miklos (2016-06-16). "[GIT PULL overlayfs fixes for 4.7-rc3"]. https://lkml.org/lkml/2016/6/16/87. Retrieved 2018-10-05.