A long time ago, I wrote Soweli Lukin, a web proxy for Gopher and Gemini. That allowed me to do some munging of the Gopher texts and render it as HTML, which meant reflowing long lines. That’s the main reason Gopher clients don’t work for me on mobile phones.
Now, if somebody wants to crawl all of Gopher space, they can crawl Gopher… or they can use a web proxy. That’d be pretty aggravating, of course, but not inconceivable. There are, after all, lots of people not wearing masks and endangering others, I bet they’re also willing to write such crawlers.
Now look at that, for the logfiles of today and the day before:
root@sibirocobombus:~# grep soweli-lukin /var/log/apache2/access.log /var/log/apache2/access.log.1 | /home/alex/bin/leech-detector ip hits bandw. hits% interv. status code distrib. 184.108.40.206 14956 5K 91% -0.0s 200 (99%), 500 (0%), 502 (0%) XX.XX.XXX.XXX 629 2K 3% -5.3s 200 (97%), 502 (1%), 500 (0%) XXX.XXX.XX.XXX 433 4K 2% -1.4s 200 (99%), 502 (0%) XX.XX.X.XXX 221 7K 1% 92.9s 502 (68%), 200 (31%) …XXXX:XXXX:XXXX:XXXX 16 5K 0% 25.2s 200 (56%), 500 (25%), 304 (18%) …XX:XXX:XXX:XX:XXX:X 12 4K 0% 1756.0s 200 (100%)
Fifteen thousand hits.
Reminds me of 220.127.116.11 who was slamming Gemini space a few days ago.
Well… I used ipset to drop their packets at the firewall, for now. Let’s see whether they change it, trying to make a comeback.
While I was at it, I also checked the IP numbers with requests in the hundreds. One of them was my monitor that checks whether the site is up, the other two were spiders with such promising names as “firstname.lastname@example.org” and “SEOkicks”. Ugh. Think of all the CO₂ wasted because of this SEO-💩.