svn/view/README.md

# MASSPIE

_Mass-scanning port `25/tcp` and checking if STARTTLS is enforced and validates_

## REQUIREMENTS

- at least 100Mbit/s up/down public network pipe
- quite a few cores to handle many DNS and SSL processes
- [MASSCAN: Mass IP port scanner](https://github.com/robertdavidgraham/masscan)
- the `host` command from BIND
- OpenSSL or LibreSSL (`s_client`)

## SYSPREP

You will have to adapt the rate to your network pipe and CPU power.  This is why it's good to have some performance monitoring tools available.

Slackware

        slackpkg install bind traceroute nmap htop iftop
        sbopkg -i nload
        #ntop
        slackpkg install gcc-10 git libcap make
        git clone https://github.com/robertdavidgraham/masscan
        cd masscan/
        make >/dev/null && echo BUILT
        make install >/dev/null && echo INSTALLED
        mkdir -p /etc/masscan/
        cp -i data/exclude.conf /etc/masscan/exclude.conf.sample

Ubuntu

        apt install dnsutils traceroute nmap htop iftop nload masscan
        #ntop

Ready to go

        which masscan
        masscan -V

Don't forget to define the exclude list e.g.

        vi /etc/masscan/exclude.conf

        0.0.0.0/8
        10.0.0.0/8
        100.64.0.0/10
        127.0.0.0/8
        169.254.0.0/16
        172.16.0.0/12
        192.0.0.0/24
        192.0.0.0/29
        192.0.0.170/32
        192.0.0.171/32
        192.0.2.0/24
        192.88.99.0/24
        192.168.0.0/16
        198.18.0.0/15
        198.51.100.0/24
        203.0.113.0/24
        240.0.0.0/4
        255.255.255.255/32

## INITIAL SCAN

_as root_

        month=`date +%Y%m`
        mkdir -p ~/mass/$month/
        cd ~/mass/$month/
        screen -S $month

        ping 208.67.220.220
        htop
        iftop
        nload

        time masscan 0.0.0.0/0 -p25 --excludefile /etc/masscan/exclude.conf --rate=250000 -oG massp25.og

You might adjust the rate according to your network bandwidth, cpu power and most of all, on how long you are willing to wait e.g. 8 to 9 hours is acceptable to me.

## INSTALLATION

_as user_

Now move the scanned materials to user's home directory.  Got 11 to 13 millions?

        grep -v ^# ~/mass/$month/massp25.og | wc -l
        mv ~/mass/$month/ ~mass/
        chown -R mass:users ~mass/$month/

and switch to it

        su - mass

Then grab the Masspie scripts and start a GNU/Screen session in there

        svn checkout https://svn.osdn.net/svnroot/masspie/
        cd masspie/

        month=`date +%Y%m`
        cd ~/mass/$month/
        screen -S masspie

## IPREV CHECK

_As user and heavy CPU times_

Stress some DNS forwarder, possibly yours, or your ISP's.  This can be split across multiple servers.  Eventually switch around the order of nameservers into `resolv.conf` e.g. first server points to DNS1 and second server points to DNS2.

        vi /etc/resolv.conf

        cd ~/$month/
        ls -lF massp25.og
        #rm -rf splitted/
        #rm -f splitted/*.ptr splitted/*.weird
        ~/masspie/checkiprev.bash
        ls -lF massp25.og.ip.sort
        ls -F splitted/ip[0-9][0-9][0-9]
        tail splitted/ip099
        tail -G splitted/ip099.ptr

and check the CPU load with `htop`

## CUSTOM OPENSSL

_Exit when EHLO/STARTTLS was not advertised_

        git clone git://git.openssl.org/openssl.git
        cd openssl/
        patch -p1 < ../openssl-shut.patch

        removepkg openssl
        #keep openssl-solibs as the whole system depends on it
        mv /etc/ssl/ /etc/ssl.old/

        ./config --openssldir=/etc/ssl
        #perl configdata.pm --dump
        time make -j8 >/dev/null && echo BUILT
        time make -j8 install >/dev/null && echo INSTALLED
        ls -lF /usr/local/lib64/libssl.so
        ls -lF /usr/local/lib64/libcrypto.so
        ls -lF /usr/local/include/openssl/aes.h
        #cat /etc/ld.so.conf
        ldconfig
        update-ca-certificates

## VALID CN/SAN CHECK

        for x in `seq -w 000 999`; do ./sslcheck.bash x$x.ptr & done; unset x
        jobs

## CERTIFICATE CHECK

Grab the [latest concatenated Mozilla CA bundle](https://curl.haxx.se/docs/caextract.html)

        cd /etc/ssl/
        curl -s --remote-name --time-cond - https://curl.haxx.se/ca/cacert.pem
        sha256sum cacert.pem | tee -a cacert.pem.sha2

        #2020/01/01
        #adf770dfd574a0d6026bfaa270cb6879b063957177a991d453ff1d302c02081f  cacert.pem

Now make sure you're in position to validate certificates.

        ehlo=YOUR-IPREV
        mx=xc.nethence.com
        echo Q | /usr/local/bin/openssl -4 s_client -showcerts -verify 5 -CAfile cacert.pem -starttls smtp -name $ehlo -servername $mx -connect $mx:25 -crlf > $mx.chain.crt
        #-CApath /etc/ssl/certs
        #-brief

Note `-showcerts` helps to get the intermediate certificate here.

        /usr/local/bin/openssl crl2pkcs7 -nocrl -certfile $mx.chain.crt | openssl pkcs7 -print_certs -noout

        /usr/local/bin/openssl verify -verbose -issuer_checks -verify_return_error -CAfile cacert.pem -untrusted $mx.chain.crt -no_alt_chains -ignore_critical $mx.chain.crt
        #-CApath /etc/ssl/certs
        #-crl_download -crl_check

Note `-untrusted` helps to define the intermediate certificate for chain validation.

Then finally proceed with mass validation.

        for x in `seq -w 000 999`; do ./sslvalid.bash x$x.ptr.validcn & done; unset x
        jobs

## PTR DOMAINS

We further need to look at MX records to query correct TLSA records, and to get MX records, we need zone names.  We will not obtain all of public zones, but here's an attempt to get some using the PTRs we've collected.

        ./checkdomains.bash

## MXes & DANE

Now we can query MX against a few domains and eventually seek for DANE-enabled SMTP hosts (including PKIX-TA/EE).

        ./dane.bash

## ADDITIONAL NOTES

The second-level domain public suffixes in file `SLDs` were obtained as such

        wget https://raw.githubusercontent.com/gavingmiller/second-level-domains/master/SLDs.csv
        cut -f2 -d, SLDs.csv | sed 's/^\.//' > SLDs
        dos2unix SLDs
