Get creative with your SSH ProxyCommands
pcmd
is a small utility to make SSH ProxyCommand
more powerful. It might be
useful any time you have a ProxyCommand
that involves a lengthy or expensive
set-up or tear-down procedure.
The primary use case of pcmd
is to wrap a ProxyCommand
that might need to
perform non-trivial set-up or tear-down operations. For example, one might have
a ProxyCommand
that provisions and deletes a cloud VPS for disposable
development environments, accessible via ssh my.temporary.host
. For such a use
case, it is essential to a) not duplicate work and b) ensure that resources get
cleaned up. Without pcmd
it can be difficult to reliably meet those
requirements.
Here are some of the challenges I faced when implementing on-demand VPS
provisioning in a ProxyCommand
:
ControlMaster auto
to make SSH opportunistically re-uses existing
connections, skipping the ProxyCommand
on subsequent SSH invocations.
There's a small problem, though: what if the first connection is still in the
"set-up" phase and hasn't established a connection? It turns out that SSH
doesn't do any locking, and, in that case, it executes the ProxyCommand
twice. One will win and become the master connection, the other will log a
warning that the master already exists, and you will be paying for double the
compute you thought you were.pcmd
solves all of those problems:
pcmd
shields the underlying ProxyCommand
from interrupt and hangup
signals. When a signal is received, pcmd closes stdio and starts a grace
period timer (default 5 minutes). During this grace period, the
ProxyCommand
can perform any tear-down it needs. If the command hasn't
exited by the end of the grace period, pcmd
sends a kill signal.pcmd
copies stderr of the ProxyCommand
to a log file as well as to the
terminal. That way, while waiting for the connection, you can see any
relevant logs. Once the connection is closed, pcmd
continues copying stderr
of the command to the log file but stops copying stderr to the terminal. That
way, you don't get annoying logs dumped into the terminal when you move on to
something else.-lock
flag, pcmd
ensures only one copy of your
ProxyCommand
runs at once. If are sharing connections using
ControlMaster
, you can additionally specify -wait-for-master
. If pcmd
doesn't have a lock, it blocks waiting for the ControlMaster
to come up. As
a nicety, while waiting, it also tails the log file mentioned in 2) to stderr
so you can monitor the progress of a lengthy set-up.pcmd
is cross-compiled to many
targets, but has only been
tested on macOS and Linux (Ubuntu 18.04). Additionally, pcmd
requires the
tail
and, depending on configuration, ssh
commands to be available. These
are likely already installed on your system.
pcmd
is available for download from the releases
page. You can also use the
following snippet to install the latest release of pcmd
, adjusting the values
accordingly for your environment:
# Possible options: darwin,linux,freebsd,openbsd,netbsd
# darwin == macOS
OS=darwin
# Possible options: amd64,386,arm,arm64(linux only)
ARCH=amd64
TARGET=pcmd-$OS-$ARCH
# Download and unzip
curl -OL https://github.com/andrewhamon/pcmd/releases/latest/download/$TARGET.zip
unzip $TARGET.zip
# Copy to somewhere on your $PATH
# replace ~/bin with something appropriate for your environment
cp $TARGET/pcmd ~/bin
If you have Go installed, you can also install pcmd
using go get
:
go get github.com/andrewhamon/pcmd
The easiest way to get started is to wrap your original ProxyCommand
with
pcmd
in your SSH config. For example, if your SSH config looks like this:
Host some-host
ProxyCommand original-proxy-command --original-arg foobar
You can prefix original-proxy-command
and its arguments with pcmd
:
Host some-host
ProxyCommand pcmd original-proxy-command --original-arg foobar
This will continue to proxy as before but also ensures original-proxy-command
has adequate time after the connection closes to perform cleanup.
Included in this repo is an example bash
script
that uses doctl
to create a VPS on the fly and proxy to it. The following is
is the resulting SSH config, configured for connection sharing with
ControlMaster
. To use, you will need to ensure that the on-demand-proxy
script is downloaded and available in your $PATH
and that you have installed
doctl
.
Host ondemand.dev
User root
ProxyCommand pcmd --wait-for-master -r %r -h %h -p %p on-demand-proxy %h
ControlMaster auto
ControlPath ~/.ssh/%r@%h:%p.sock
# Keep the connection alive for 5 minutes after the last connection is closed.
# This lets you quickly re-connect. This is not the same as the -grace-period
# flag.
ControlPersist 300
# Even when reviving a snapshot, DigitalOcean seems to generate a new key, which
# will then cause scary warnings.
StrictHostKeyChecking no
UserKnownHostsFile=/dev/null
You can configure the grace period (default 5 minutes) with the -grace-period
flag. For example, to set the grace period to 10 minutes:
pcmd -grace-period 600 original-proxy-command --original-arg foobar
The grace period begins only once the SSH connection is closed.
ProxyCommand
invocationsIf you add the -lock
flag, pcmd
can ensure that only one copy of
original-proxy-command
runs at a time. To do so, pcmd
needs to know the SSH
remote user and host, which it uses to form a unique key for locking. For
example:
pcmd -lock -r %r -h %h original-proxy-command --original-arg foobar
%r
and %h
are expanded by SSH automatically. See TOKENS in SSH_CONFIG(5) for
more details.
SSH provides a native mechanism for connection sharing via the ControlMaster
and ControlPath
configuration options, which allows you to share a single
connection for multiple SSH sessions (see SSH_CONFIG(5) for more details). One
issue with the native mechanism, however, is that SSH doesn't do any concurrency
control. That means if you have a lengthy set-up in your proxy command, and then
try to establish two concurrent connections, SSH does not block one connection
waiting for a master. Instead, it runs both ProxyCommand
s at the same time.
pcmd
can help with this, by doing two things:
flock
system call) to ensure only one version
of your ProxyCommand
is running at a time.ControlMaster
to come up, if the lock can not be acquired.
pcmd
checks the status of the control master using ssh user@host -O check
.To set up connection sharing, add the -wait-for-master
flag, along with the
-r
and -h
flags. For example:
Host some-host
ProxyCommand pcmd -wait-for-master -r %r -h %h original-proxy-command --original-arg foobar
ControlMaster auto
ControlPath ~/.ssh/ssh-%r@%h:%p
# Setting ControlPersist to a non-zero value ensures that SSH keeps the master
# connection running in the background, rather than blocking the first SSH
# invocation until all child connections also complete. If you want to keep your
# connection alive in the background for longer, set this to a higher value (in
# number of seconds)
ControlPersist 1
The above config will allow only a single master connection, and make any subsequent connections wait for the master to come up before continuing.