Everyone with a public IP knows the SSH brute-force noise is constant. You harden the box, you move on. But I have always wanted to do something more entertaining than just dropping the packets, so this week I put a tarpit on port 22 and pointed real SSH somewhere else. A tarpit does not block the bots. It holds them. And watching them sit there stuck for hours is the most fun I have had with a firewall rule in ages.

The tool is endlessh. Here is how I set it up on the edge box, what it does under the bonnet, and what a week of logs actually showed.

Why a tarpit works at all

When an SSH client connects, the server is allowed to send a banner of text before the real protocol handshake begins. The RFC puts no hard limit on how many lines come before the SSH-2.0 identification string. endlessh abuses that politely: it accepts the connection and then dribbles out an endless, random banner a few bytes at a time, forever. The scanner sits there waiting for a handshake that never arrives, holding a socket and a thread open on the attacker's side for as long as it is willing to wait. You spend almost nothing. They spend a connection slot. Multiply that across a botnet and you are wasting a measurable slice of someone's scanning capacity for the price of a tiny daemon.

Move real SSH out of the way first

Do this before anything else or you will lock yourself out. Real sshd goes to a high port, reachable only from the LAN and the VPN. Edit /etc/ssh/sshd_config:

Port 2222
ListenAddress 10.0.0.1
PermitRootLogin no
PasswordAuthentication no

Keys only, no passwords, and it is not even listening on the WAN interface. Restart it and confirm you can still get in on the new port from inside before you touch the firewall. I cannot stress that enough. I have locked myself out of the rack twice over the years and both times it was a two-flight walk to the cupboard with a keyboard.

Install and configure endlessh

It is a single small daemon. On a Debian-based edge box it is packaged, or you build the C source in about ten seconds. The config lives at /etc/endlessh/config:

Port 22
Delay 10000
MaxLineLength 32
MaxClients 4096
LogLevel 1

Delay is the milliseconds between each dribble of bytes, so ten seconds here. MaxLineLength keeps each junk line short so you are sending almost nothing. MaxClients caps how many trapped connections you hold at once so a flood cannot exhaust your own file descriptors. LogLevel 1 logs every accept and every close, which is the whole point, because the logs are the fun part.

Binding to port 22 needs a privilege. Rather than run the thing as root, I give the service just the one capability it needs. The systemd unit gets:

[Service]
ExecStart=/usr/bin/endlessh
AmbientCapabilities=CAP_NET_BIND_SERVICE
PrivateUsers=true
DynamicUser=yes
Restart=on-failure

That lets an unprivileged, dynamically-allocated user bind the low port and nothing else. If endlessh is ever popped, and it is a tiny attack surface, it is a nobody user with no capabilities beyond binding a socket it already holds.

The firewall side

Port 22 is now open to the world and points at the tarpit. Real SSH on 2222 is restricted to the trusted networks. With nftables:

table inet filter {
  chain input {
    type filter hook input priority 0; policy drop;
    ct state established,related accept
    iif "lo" accept
    tcp dport 22 accept
    ip saddr 10.0.0.0/24 tcp dport 2222 accept
  }
}

Port 22 accepted from anywhere and handed to endlessh; 2222 accepted only from the LAN. Everything else dropped. Note I am not doing a redirect here, endlessh is simply listening on 22 directly, which is one fewer moving part than a DNAT rule and easier to reason about at 2am.

What a week of logs showed

endlessh logs a line on accept and a line on close with the total time trapped. A week of journalctl -u endlessh on a single residential IP with nothing interesting behind it:

  • Roughly nine thousand connections from around six hundred unique source addresses. For a boring home IP that is just the background radiation of the internet.
  • The median trap time was a few minutes, but the long tail is the good bit. A handful of clients stayed connected for over three hours before giving up, and two came back and got stuck again the next day, which suggests nobody at the other end was watching either.
  • The source addresses were the usual mix of compromised boxes on residential and cloud ranges. I do not bother blocking them individually. The tarpit is the response.

None of this is a defence on its own, to be clear. The real SSH port being off the WAN and key-only is what protects the box. The tarpit is what makes the leftover noise entertaining instead of irritating, and it costs a rounding error of CPU to run. If your edge box is a Pi or an old thin client sitting idle most of the day, this is a good use of it.

The source and full option list are at the endlessh repository, and if you want to understand exactly which part of the handshake it is abusing, the banner behaviour is defined in RFC 4253. Move your real SSH first, then go and waste some bot time.