Blocking complete ASN IP ranges automatically on FreeBSD with pf based on nginx logs

Some time ago I though that putting up an AI poison maze on my site is a great way of doing something against bots that do not respect robots.txt. It was, until I got hit by botnets which never back off.

Initially, it was a syn flood. I don't know why I believed that the FreeBSD and pf defaults are protected against syn flood, so there was a bit of tweaking needed in `/etc/sysctl.conf. Many thanks for the internet sources12.

# drop tcp packets destined for closed ports (default 0) 
net.inet.tcp.blackhole=2
# SYN/FIN packets get dropped on initial connection (default 0)
net.inet.tcp.drop_synfin=1
# recycle FIN/WAIT states quickly, helps against DoS, but may cause false RST (default 0)
net.inet.tcp.fast_finwait2_recycle=1
# disable TCP Fast Open client side, enforce three way TCP handshake (default 1, enabled)
net.inet.tcp.fastopen.client_enable=0
# disable TCP Fast Open server side, enforce three way TCP handshake (default 0)
net.inet.tcp.fastopen.server_enable=0
# TCP FIN_WAIT_2 timeout waiting for client FIN packet before state close (default 60000, 60 sec)
net.inet.tcp.finwait2_timeout=1000
# icmp may not send RST to avoid spoofed icmp/udp floods (default 1)
net.inet.tcp.icmp_may_rst=0
# amount of tcp keep alive probe failures before socket is forced closed (default 8)
net.inet.tcp.keepcnt=2
# time before starting tcp keep alive probes on an idle, TCP connection (default 7200000, 7200 secs)
net.inet.tcp.keepidle=62000
# tcp keep alive client reply timeout (default 75000, 75 secs)
net.inet.tcp.keepinit=5000
# Maximum Segment Lifetime, time the connection spends in TIME_WAIT state (default 30000, 2*MSL = 60 sec)
net.inet.tcp.msl=2500
# disable for mtu=1500 as most paths drop ICMP type 3 packets, but keep enabled for mtu=9000 (default 1)
net.inet.tcp.path_mtu_discovery=0

But it just kept going, and nepenthes3 got overwhelmed.

At first, I turned off nepenthes, and returned a mere HTTP 410 from nginx directly.

And it kept going, because now the logging was so high traffic that fail2ban was consuming enough CPU that everything else was starting to have problem. I didn't think this was possible, but yes: apparently fail2ban can use quite a lot of resources when it gets combined with high volume logging.

At one point I added a tarpit directly inside nginx, using the echo module4 and it's delay feature.

    echo '    <!DOCTYPE html>';
    echo_sleep 2;
    echo '    <html lang=en>';
    echo_sleep 4;
    echo '        <head>';
    echo_sleep 8;
    echo '            <meta charset=utf-8>';
    echo_sleep 16;
    echo '            <title></title>';
    echo_sleep 32;
    echo '        </head>';
    echo_sleep 64;
    echo '        <body>';
    echo_sleep 128;
    echo '            <blockquote>';
    echo_sleep 256;
    echo '                <p>';
    echo_sleep 512;
    echo '                    I must not fear. ';
    echo_sleep 1024;
    echo '                    Fear is the mind-killer. ';
    echo_sleep 2048;
    echo '                    Fear is the little-death that brings total obliteration. ';
    echo_sleep 4096;
    echo '                    I will face my fear. ';
    echo_sleep 8192;
    echo '                    I will permit it to pass over me and through me. ';
    echo_sleep 16384;
    echo '                    And when it has gone past I will turn the inner eye to see its path. ';
    echo_sleep 32768;
    echo '                    Where the fear has gone there will be nothing. ';
    echo_sleep 65535;
    echo '                    Only I will remain.';
    echo_sleep 131072;
    echo '                </p>';
    echo_sleep 262144;
    echo '                <p>—Frank Herbert, <cite>Dune</cite></p>';
    echo_sleep 524288;
    echo '            </blockquote>';
    echo_sleep 1048576;
    echo '        </body>';
    echo_sleep 2097152;
    echo '    </html>';

I ended up with 130k open connection. harold

The current solution, which I'm hoping to work for a while, was to add ASN data into my blocked logs, devise a cron job that gets the list of ASN entries, and block them if they go above X per hour - X is currently 100. So far, this is working.

Step 0: get ASN data

/usr/local/etc/cron.d/root

  • root, because I need to reload nginx
  • yes, I use wget
  • yes, my crons have bash as shell
@daily root wget -q -O/var/db/ip-location-db/geolite2-asn.mmdb https://github.com/sapics/ip-location-db/releases/download/latest/geolite2-asn.mmdb >/dev/null && /usr/sbin/service nginx reload

@daily root cd /var/db/ip-location-db && wget -O/var/db/ip-location-db/asn_full.json.zip https://geoip.oxl.app/file/asn_full.json.zip && unzip -d /var/db/ip-location-db -f /var/db/ip-location-db/asn_full.json.zip && rm /var/db/ip-location-db/asn_full.json.zip && /usr/sbin/service nginx reload

Step 2: mount a tmpfs, so I don't wear out my SSD

While tmpfs is linuxism, but in this case, it's useful. There's a log rotation set up for 200MB max size + daily, keeping only 1 additional, so it can't eat my memory, but I have little desire to wear out my SSD with these logs.

/etc/fstab

tmpfs /shm tmpfs rw,mode=1777 0 0

/boot/loader.conf

tmpfs_load="YES"

If you don't want to reboot:

kldload tmpfs

Step 3: get ANS data into nginx logs5

Install the nginx geoip2 module - I leave it to you if you compile nginx from ports with the relevant flags or install nginx-full, both are OK. I went with the first.

nginx.conf

load_module /usr/local/libexec/nginx/ngx_http_geoip2_module.so;

http {
        [...]

    geoip2 /var/db/ip-location-db/geolite2-asn.mmdb {
        $geoip2_asn autonomous_system_number;
        $geoip2_asname autonomous_system_organization;
    }

    log_format minimal '$time_iso8601 $remote_addr $geoip2_asn "$geoip2_asname" $server_name "$request" $status $request_time';
    
    [...]
    
    server {
      [...]
      location /nope {
          access_log /shm/blocked.log minimal;
          return 401;
      }
    }
}

Step 4: block all the things

/usr/local/bin/block-asn-pf.py

#!/usr/local/bin/python3.11

import argparse
import json
import syslog
import os

parser = argparse.ArgumentParser()
parser.add_argument('asn')
args = parser.parse_args()

with open('/var/db/ip-location-db/asn_full.json') as f:
    asninfo = json.load(f)

for cidr in asninfo.get(args.asn, {}).get('ipv4', []):
    cmd=f"pfctl -t badips -T add {cidr}"
    os.system(cmd)

/usr/local/etc/cron.d/root

  • root, because it pipes into pf
  • there's most likely an elegant way to do it in awk only, but there's a reason why it's called awkward
@hourly root for asn in $(cat /shm/blocked.log| awk '{print $3}' | sort | uniq -cd | awk -v limit=100 '$1 > limit{print $2}'); do /usr/local/bin/block-asn-pf.py $asn >/dev/null 2>&1; done;

# clean IPs up after a day
@hourly root pfctl -t badips -T expire 86400 >/dev/null 2>&1

/usr/local/etc/pf.conf

table <badips> persist
block drop in quick from <badips> to any 
block in from <badips>

Extra: block Spamhaus DROP

Spamhaus provides a "Don't Route Or Peer Lists (DROP)" 6 list for IP addresses that are really not nice, so it's worth adding them to the same blocked table

/usr/local/etc/cron.d/root

  • root, it pipes into pf
@hourly root curl -s https://www.spamhaus.org/drop/drop_v4.json | jq -r '.cidr' | grep -E '^[0-9]' |  sed -r 's/(.*)/pfctl -t badips -T add \1/' | bash >/dev/null 2>&1

  1. https://forums.freebsd.org/threads/trying-to-fight-syn-flood.76879/↩︎

  2. https://calomel.org/freebsd_network_tuning.html↩︎

  3. https://zadzmo.org/code/nepenthes/↩︎

  4. https://github.com/openresty/echo-nginx-module#echo_sleep↩︎

  5. https://www.bakker.net/kb/nginx-geoip/↩︎

  6. https://www.spamhaus.org/blocklists/do-not-route-or-peer/↩︎

This entry was written by Peter Molnar, and originally posted on petermolnar dot net.

⋅ CC-BY-4.0 ⋅ ⋅ https://petermolnar.net/article/asn-blocking-freebsd-pf-nginx/