Administration

These are the pages for all my system administration woes. I rarely post happy things in this section. You have been warned.

2020-04-05 Jitsi Considerations

Here are some things to look out for:

  1. Does your Jitsi instance use things that enable tracking, such as Google Analytics, Google Fonts, Google STUN servers?
  2. Does your Jitsi instance have a decent privacy policy, such as minimal logging, no sharing with third parties, and so on?
  3. If you’re self-hosting, have you considered these issues?
  4. Make sure people only use the apps, or Chrome – apparently Safari doesn’t work and Firefox ruins the experience for everybody else in the meeting
  5. Understand that all the apps except for the one on the F-Droid store for Android come with trackers

This is based on the following sources:

  1. Good info by Freifunk München on how to use Jitsi, and a link to their instance
  2. Also, an app for the desktop based on Electron, so at least you don’t have to run Google Chrome
  3. Privacy Tools on Zoom vs. Jitsi, listing some of the issues with Jitsi

Tags:

Comments on 2020-04-05 Jitsi Considerations

I installed the Debian package for the Freifunk Jitsi app and that worked just fine.

alex@melanobombus:~$ sudo dpkg -i Downloads/ffmuc-meet-amd64.deb 
[sudo] password for alex: 
Selecting previously unselected package jitsi-meet-electron.
(Reading database ... 498341 files and directories currently installed.)
Preparing to unpack Downloads/ffmuc-meet-amd64.deb ...
Unpacking jitsi-meet-electron (2.0.4-31) ...
dpkg: dependency problems prevent configuration of jitsi-meet-electron:
 jitsi-meet-electron depends on libappindicator3-1; however:
  Package libappindicator3-1 is not installed.

dpkg: error processing package jitsi-meet-electron (--install):
 dependency problems - leaving unconfigured
Processing triggers for gnome-menus (3.31.4-3) ...
Processing triggers for mime-support (3.62) ...
Processing triggers for desktop-file-utils (0.23-4) ...
Processing triggers for hicolor-icon-theme (0.17-2) ...
Errors were encountered while processing:
 jitsi-meet-electron
alex@melanobombus:~$ apt search libappindicator3-1
Sorting... Done
Full Text Search... Done
gir1.2-appindicator3-0.1/amber 0.4.92-7 amd64
  Typelib files for libappindicator3-1

libappindicator3-1/amber 0.4.92-7 amd64
  allow applications to export a menu into the panel -- GTK3 version

alex@melanobombus:~$ sudo apt install libappindicator3-1
Reading package lists... Done
Building dependency tree       
Reading state information... Done
You might want to run 'apt --fix-broken install' to correct these.
The following packages have unmet dependencies:
 libappindicator3-1 : Depends: libindicator3-7 but it is not going to be installed
E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution).
alex@melanobombus:~$ sudo apt --fix-broken install
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Correcting dependencies... Done
The following additional packages will be installed:
  libappindicator3-1 libindicator3-7
The following NEW packages will be installed:
  libappindicator3-1 libindicator3-7
0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
1 not fully installed or removed.
Need to get 106 kB of archives.
After this operation, 217 kB of additional disk space will be used.
Do you want to continue? [Y/n] 
Get:1 https://repo.pureos.net/pureos amber/main amd64 libindicator3-7 amd64 0.5.0-4 [52.9 kB]
Get:2 https://repo.pureos.net/pureos amber/main amd64 libappindicator3-1 amd64 0.4.92-7 [53.5 kB]
Fetched 106 kB in 1s (187 kB/s)               
Selecting previously unselected package libindicator3-7:amd64.
(Reading database ... 498429 files and directories currently installed.)
Preparing to unpack .../libindicator3-7_0.5.0-4_amd64.deb ...
Unpacking libindicator3-7:amd64 (0.5.0-4) ...
Selecting previously unselected package libappindicator3-1:amd64.
Preparing to unpack .../libappindicator3-1_0.4.92-7_amd64.deb ...
Unpacking libappindicator3-1:amd64 (0.4.92-7) ...
Setting up libindicator3-7:amd64 (0.5.0-4) ...
Setting up libappindicator3-1:amd64 (0.4.92-7) ...
Setting up jitsi-meet-electron (2.0.4-31) ...
Processing triggers for libc-bin (2.28-10) ...

OK, there was this little problem because I didn’t know how to install a package from a file and satisfy all its dependency from the regular repos.

Also, I’m starting to feel like installing all this stuff from packages I download myself is a security nightmare. These packages are never going to get fixed! 😟

– Alex Schroeder 2020-04-05 12:14 UTC

Add Comment

2020-03-26 Jitsi and the CPU

Yesterday I ran my RPG game on the self-hosted Jitsi installation. All participants need to have the right bandwidth, of course. If you’re close to Zürich, you probably have glass fibre, so no problem. As it turns out, CPU seems to be the limiting factor. The machine I’m renting has two cores and 6GB RAM available. Here are the Munin reports:

Firewall throughput CPU usageLoad average Memory usage

Tags:

Comments on 2020-03-26 Jitsi and the CPU

Other people: Jitsi Meet and the Number of Video Participants – 「… 2 virtual cores and 4GB of memory … both CPUs ran at around 70% with 10 clients and memory use was around 1.4 GBs at the beginning and 800 MB a little while later when everyone had joined and settled in … I was far away from saturating the CPU with 10 simultaneous video streams in the conference.」 – Nice! 🙂

Other software: Big Blue Button: Requirements – 「… we recommend starting with a ‘clean’ Ubuntu 16.04 64-bit server dedicated for BigBlueButton. … 4 GB of memory with swap enabled (8 GB of memory is better) … 4 CPU cores (8 is better)」 – but as noted I only have 2 GB of memory, only 2 CPU cores, it’s not a dedicated server, and it’s Debian Buster, not Ubuntu 16. Oh well!

– Alex Schroeder 2020-03-28 11:54 UTC


Had another session yesterday and when the conference ended it seemed to me that some of the other services on this machine had been killed. Out of memory? I don’t think so. I need to investigate. Perhaps the response times for requests had grown so long that Monit decided the services were down and started restarting them?

– Alex Schroeder 2020-03-29 10:06 UTC

Add Comment

2020-03-18 Selfhosting Jitsi

I’m trying to self-host Jitsi (”multi-platform open-source video conferencing”).

  1. follow the instructions and use their Ubuntu/Debian repository
  2. I already have the subdomain chat.campaignwiki.org setup with Gandi
  3. add chat.campaignwiki.org to the existing Campaign Wiki domain names in /etc/dehydrated/domains.txt such that Let’s Encrypt will pick it up
  4. edit the Apache config file they generate and comment their settings for SSLProtocol, SSLCipherSuite and SSLHonorCipherOrder, preferring my own
  5. enable the site using a2ensite and restart Apache gracefully using systemctl reload apache2

Testing:

  1. chat.campaignwiki.org seems to work!
  2. got to Firefox preferences and search for Microphone and Camera and make sure to allow requests asking to access them (which I have disabled by default)
  3. on the phone, install the Jitsi Meet app (not sure why the browser isn’t good enough on the phone?)
  4. use the test room to test it...

I get the rooms listed, but I am immediately disconnected when accessing it on the phone. Not good!

I also get disconnected every now and then when using the website on my laptop. Hm.

OK, opening two private browsing windows and connecting to a channel on the official site seems to work. At least I get two copies of my camera showing video and I can chat with myself. It also works with the app on the phone. Thus, this is not an app or browser problem.

Back to my own installation. Let’s check the logs at /var/log/jitsi/jicofo.log. Some things to note after a restart:

  1. “Lonely participants will be “terminated” after 20000 milliseconds” → that would explain my automatic disconnects on the laptop after a bit
  2. the XMPP component says “This server does not serve auth.campaignwiki.org” but I think that’s OK?
  3. I also see references to jitsi-videobridge.campaignwiki.org and focus.campaignwiki.org but surely this cannot work?
  4. there are other Java exceptions, too; these look concerning: “ClassNotFoundException: javax.activation.DataSource”, “ClassNotFoundException: javax.xml.bind.PropertyException”, and “ClassNotFoundException: javax.xml.bind.Unmarshaller”

I think I need to redo my setup. Perhaps the domain name setup was a problem? I remember providing the wrong domain name at some point and then just fixing it in the Apache site config.

  1. apt purge jigasi jitsi-meet jitsi-meet-web-config jitsi-meet-prosody jitsi-meet-web jicofo jitsi-videobridge
  2. apt install -y jitsi-meet
  3. domain to use: chat.campaignwiki.org
  4. use my own certificates: /var/lib/dehydrated/certs/campaignwiki.org/privkey.pem and /var/lib/dehydrated/certs/campaignwiki.org/cert.pem

The differences in the Apache site setup:

--- /etc/apache2/sites-available/chat.campaignwiki.org.conf
+++ /etc/apache2/sites-available/600-chat.campaignwiki.org.conf
@@ -11,13 +11,14 @@
 
   ServerName chat.campaignwiki.org
 
-  SSLProtocol TLSv1 TLSv1.1 TLSv1.2
+  # SSLProtocol TLSv1 TLSv1.1 TLSv1.2
   SSLEngine on
   SSLProxyEngine on
-  SSLCertificateFile /var/lib/dehydrated/certs/campaignwiki.org/cert.pem
-  SSLCertificateKeyFile /var/lib/dehydrated/certs/campaignwiki.org/privkey.pem
-  SSLCipherSuite "EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA256:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384:EDH+aRSA+AESGCM:EDH+aRSA+SHA256:EDH+aRSA:EECDH:!aNULL:!eNULL:!MEDIUM:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS:!RC4:!SEED"
-  SSLHonorCipherOrder on
+  SSLCertificateFile      /var/lib/dehydrated/certs/campaignwiki.org/cert.pem
+  SSLCertificateKeyFile   /var/lib/dehydrated/certs/campaignwiki.org/privkey.pem
+  SSLCertificateChainFile /var/lib/dehydrated/certs/campaignwiki.org/chain.pem
+  # SSLCipherSuite "EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA256:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384:EDH+aRSA+AESGCM:EDH+aRSA+SHA256:EDH+aRSA:EECDH:!aNULL:!eNULL:!MEDIUM:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS:!RC4:!SEED"
+  # SSLHonorCipherOrder on
   Header set Strict-Transport-Security "max-age=31536000"
 
   DocumentRoot "/usr/share/jitsi-meet"

Diff finished.  Wed Mar 18 10:33:44 2020

I still get the same problem:

  1. https://chat.campaignwiki.org/Test loads
  2. I see the UI
  3. I am reconnected after a few seconds

Image 1

Trying to see whether the hostnames make more sense, now:

# grep campaignwiki /etc/prosody/conf.d/chat.campaignwiki.org.cfg.lua 
VirtualHost "chat.campaignwiki.org"
                key = "/etc/prosody/certs/chat.campaignwiki.org.key";
                certificate = "/etc/prosody/certs/chat.campaignwiki.org.crt";
Component "conference.chat.campaignwiki.org" "muc"
admins = { "focus@auth.chat.campaignwiki.org" }
Component "jitsi-videobridge.chat.campaignwiki.org"
VirtualHost "auth.chat.campaignwiki.org"
        key = "/etc/prosody/certs/auth.chat.campaignwiki.org.key";
        certificate = "/etc/prosody/certs/auth.chat.campaignwiki.org.crt";
Component "focus.chat.campaignwiki.org"

These subdomains make my head hurt. How can this work? I have added the four of them to my /etc/hosts file, but no effect. Not surprisingly, it turns out that these need to be “real” domains, reachable from the outside. Continue reading the comments...

Tags:

Comments on 2020-03-18 Selfhosting Jitsi

What domain do you have in /etc/jitsi/meet/*-config.js ? Search the wrong domain with ack --js /etc or something. Frankly, it might be faster to redo the install from scratch on a new machine: it shouldn’t take more than 5-10 min, if you have all the ports open and there’s no other webserver installed on the same machine.

– Federico 2020-03-18 11:02 UTC


I have already redone the config once, so I don’t mind doing it again, but I need to understand what went wrong because the problem seems to be unchanged.

I checked the config file:

# grep campaignwiki /etc/jitsi/meet/chat.campaignwiki.org-config.js 
        domain: 'campaignwiki.org',
        // authdomain: 'campaignwiki.org',
        // jirecon: 'jirecon.campaignwiki.org',
        // call_control: 'callcontrol.campaignwiki.org',
        // focus: 'focus.campaignwiki.org',
        muc: 'conference.campaignwiki.org'
    bosh: '//campaignwiki.org/http-bind',
    // focusUserJid: 'focus@auth.campaignwiki.org',
    //     // 'https://campaignwiki.org/static/oauth.html'
    //          'https://campaignwiki.org/subfolder/static/oauth.html'

This is strange, because the “chat” subdomain seems to be missing. I’ll add that and try again:

--- /etc/jitsi/meet/chat.campaignwiki.org-config.js~
+++ /etc/jitsi/meet/chat.campaignwiki.org-config.js
@@ -6,7 +6,7 @@
 
     hosts: {
         // XMPP domain.
-        domain: 'campaignwiki.org',
+        domain: 'chat.campaignwiki.org',
 
         // When using authentication, domain for guest users.
         // anonymousdomain: 'guest.example.com',
@@ -28,7 +28,7 @@
     },
 
     // BOSH URL. FIXME: use XEP-0156 to discover it.
-    bosh: '//campaignwiki.org/http-bind',
+    bosh: '//chat.campaignwiki.org/http-bind',
 
     // The name of client node advertised in XEP-0115 'c' stanza
     clientNode: 'http://jitsi.org/jitsimeet',

Diff finished.  Wed Mar 18 12:26:23 2020

Sadly, that doesn’t work either. Now I’m getting an “Unfortunately, something went wrong” error.

– Alex Schroeder 2020-03-18 11:20 UTC


OK, next attempt. I found this in the Juno Host docs:

Jitsi requires you create additionals domains in your DNS and in your YunoHost

  • auth.jitsi.domain.tld
  • conference.jitsi.domain.tld
  • jitsi-videobridge.jitsi.domain.tld
  • focus.jitsi.domain.tld

So I created them:

chat 10800 IN A 178.209.50.237
chat 10800 IN AAAA 2a02:418:6a04:178:209:50:237:1
chat.auth 10800 IN A 178.209.50.237
chat.auth 10800 IN AAAA 2a02:418:6a04:178:209:50:237:1
chat.conference 10800 IN A 178.209.50.237
chat.conference 10800 IN AAAA 2a02:418:6a04:178:209:50:237:1
chat.focus 10800 IN A 178.209.50.237
chat.focus 10800 IN AAAA 2a02:418:6a04:178:209:50:237:1
chat.jitsi-videobridge 10800 IN A 178.209.50.237
chat.jitsi-videobridge 10800 IN AAAA 2a02:418:6a04:178:209:50:237:1

And the config file:

--- /etc/jitsi/meet/chat.campaignwiki.org-config.js~
+++ /etc/jitsi/meet/chat.campaignwiki.org-config.js
@@ -6,7 +6,7 @@
 
     hosts: {
         // XMPP domain.
-        domain: 'campaignwiki.org',
+        domain: 'chat.campaignwiki.org',
 
         // When using authentication, domain for guest users.
         // anonymousdomain: 'guest.example.com',
@@ -24,11 +24,11 @@
         // focus: 'focus.campaignwiki.org',
 
         // XMPP MUC domain. FIXME: use XEP-0030 to discover it.
-        muc: 'conference.campaignwiki.org'
+        muc: 'conference.chat.campaignwiki.org'
     },
 
     // BOSH URL. FIXME: use XEP-0156 to discover it.
-    bosh: '//campaignwiki.org/http-bind',
+    bosh: '//chat.campaignwiki.org/http-bind',
 
     // The name of client node advertised in XEP-0115 'c' stanza
     clientNode: 'http://jitsi.org/jitsimeet',

Diff finished.  Wed Mar 18 21:12:06 2020

And now a single user can connect! I can connect to a chatroom and I see my video. No constant reconnects. I also had to restart Prosody with service prosody reload.

The only problem now is that whenever I connect to the same chatroom from a different private window or the phone, both sessions are terminated.

Perhaps this is still related to certificates? I think the one I have is only good for campaignwiki.org, www.campaignwiki.org, and chat.campaignwiki.org – but I need more subdomains, apparently.

– Alex Schroeder 2020-03-18 20:21 UTC


I decided to use the prosody cert calls mentioned in the Configure Prosody section. (Now I remember how I failed to set it up a while ago.)

Anyway:

prosodyctl cert generate chat.campaignwiki.org
prosodyctl cert generate auth.chat.campaignwiki.org
systemctl restart prosody

OK... I can now connect with both browser and phone, and we see each other in the channel, but both believe the other is muted and has no video!

Gaaaah.

– Alex Schroeder 2020-03-18 20:32 UTC


OK, /var/log/jitsi/jicofo.log has a SEVERE message: “... XMPP reconnection failed: ... SSLHandshakeException: PKIX path validation failed: ... CertPathValidatorException: signature check failed”

Let’s try this:

ln -sf /var/lib/prosody/auth.chat.campaignwiki.org.crt /usr/local/share/ca-certificates/
update-ca-certificates -f
systemctl restart prosody
systemctl restart jicofo
systemctl restart jitsi-videobridge

– Alex Schroeder 2020-03-18 20:48 UTC


Oh wow, it works! 😀

– Alex Schroeder 2020-03-18 20:50 UTC


If you’re using the app on your phone, note the trackers!

– Alex Schroeder 2020-03-18 21:21 UTC


Oh, cool:

– Alex Schroeder 2020-03-19 11:51 UTC


Some stuff is still not correct. Jitsi should give me a working XMPP server, too, as far as I understand it. Right now, however, Bitlbee gives me an error: “Login error: Certificate verification problem 0x14: certificate hasn’t got a known issuer”. That isn’t too surprising, considering that I self-generated certificates and added one of them to the local trust db. I guess the answer is to generate a certificate using Let’s Encrypt for all the domains in question.

I edited /etc/dehydrated/domains.txt and now I’m listing these:

campaignwiki.org www.campaignwiki.org chat.campaignwiki.org auth.chat.campaignwiki.org

When I run dehydrated -c I get an error:

 + Responding to challenge for auth.chat.campaignwiki.org authorization...
 + Hook: Nothing to do...
 + Cleaning challenge tokens...
 + Hook: Nothing to do...
 + Challenge validation has failed :(
ERROR: Challenge is invalid! (returned: invalid) (result: {
  "type": "http-01",
  "status": "invalid",
  "error": {
    "type": "urn:ietf:params:acme:error:dns",
    "detail": "DNS problem: NXDOMAIN looking up A for auth.chat.campaignwiki.org - check that a DNS record exists for this domain",
    "status": 400
  },
  "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/3446546602/NNlp-g",
  "token": "l7Q3SpNFqQgZT3lnVVEPps0DvN1NkoyfG5rhC_dEMwY"
})

When I run dig on campaignwiki.org, www.campaignwiki.org, chat.campaignwiki.org and auth.chat.campaignwiki.org I do get some output but it doesn’t look correct:

;; QUESTION SECTION:
;auth.chat.campaignwiki.org.	IN	A

;; AUTHORITY SECTION:
campaignwiki.org.	2776	IN	SOA	ns1.gandi.net. hostmaster.gandi.net. 1584556785 10800 3600 604800 10800

Compare with the following:

;; QUESTION SECTION:
;chat.campaignwiki.org.		IN	A

;; ANSWER SECTION:
chat.campaignwiki.org.	3216	IN	A	178.209.50.237

So the answer section doesn’t include the subdomain. How strange.

Actually...

;; QUESTION SECTION:
;chat.auth.campaignwiki.org.	IN	A

;; ANSWER SECTION:
chat.auth.campaignwiki.org. 3600 IN	A	178.209.50.237

I think I got it backwards! 😟

Let’s try the following in my DNS setup:

chat 10800 IN A 178.209.50.237
chat 10800 IN AAAA 2a02:418:6a04:178:209:50:237:1
auth.chat 10800 IN A 178.209.50.237
auth.chat 10800 IN AAAA 2a02:418:6a04:178:209:50:237:1
conference.chat 10800 IN A 178.209.50.237
conference.chat 10800 IN AAAA 2a02:418:6a04:178:209:50:237:1
focus.chat 10800 IN A 178.209.50.237
focus.chat 10800 IN AAAA 2a02:418:6a04:178:209:50:237:1
jitsi-videobridge.chat 10800 IN A 178.209.50.237
jitsi-videobridge.chat 10800 IN AAAA 2a02:418:6a04:178:209:50:237:1

– Alex Schroeder 2020-03-19 12:34 UTC


OK, fixing stuff, now that the subdomains and subsubdoains 🙂 are ready:

rm /usr/local/share/ca-certificates/auth.chat.campaignwiki.org.crt
rm /etc/prosody/certs/*campaign*
for subdomain in chat auth.chat focus.chat jitsi-videobridge.chat conference.chat; do
    cat /var/lib/dehydrated/certs/campaignwiki.org/fullchain.pem > /etc/prosody/certs/$subdomain.campaignwiki.org.crt
    cat /var/lib/dehydrated/certs/campaignwiki.org/privkey.pem > /etc/prosody/certs/$subdomain.campaignwiki.org.key
done
chown prosody.prosody /etc/prosody/certs/*campaign*
update-ca-certificates -f
systemctl restart prosody
systemctl restart jicofo
systemctl restart jitsi-videobridge

So now I’m expecting the following:

# ls -l /etc/prosody/certs
total 40
-rw-r--r-- 1 prosody prosody 4006 Mar 19 22:10 auth.chat.campaignwiki.org.crt
-rw-r--r-- 1 prosody prosody 3243 Mar 19 22:10 auth.chat.campaignwiki.org.key
-rw-r--r-- 1 prosody prosody 4006 Mar 19 22:10 chat.campaignwiki.org.crt
-rw-r--r-- 1 prosody prosody 3243 Mar 19 22:10 chat.campaignwiki.org.key
-rw-r--r-- 1 prosody prosody 4006 Mar 19 22:10 conference.chat.campaignwiki.org.crt
-rw-r--r-- 1 prosody prosody 3243 Mar 19 22:10 conference.chat.campaignwiki.org.key
-rw-r--r-- 1 prosody prosody 4006 Mar 19 22:10 focus.chat.campaignwiki.org.crt
-rw-r--r-- 1 prosody prosody 3243 Mar 19 22:10 focus.chat.campaignwiki.org.key
-rw-r--r-- 1 prosody prosody 4006 Mar 19 22:10 jitsi-videobridge.chat.campaignwiki.org.crt
-rw-r--r-- 1 prosody prosody 3243 Mar 19 22:10 jitsi-videobridge.chat.campaignwiki.org.key
lrwxrwxrwx 1 root    root      37 Jan  9  2019 localhost.crt -> ../../ssl/certs/ssl-cert-snakeoil.pem
lrwxrwxrwx 1 root    root      39 Jan  9  2019 localhost.key -> ../../ssl/private/ssl-cert-snakeoil.key

As you can see, the various certificates are all identical because I’ve setup dehydrated such that the one certificate works for all these names. This differs from the official Let's Encrypt and Prosody documentation!

Once these certificates are in place and have the correct owners, I can edit my hook.sh for dehydrated:

# cat /var/lib/dehydrated/hook.sh
#!/bin/bash
if [ ${1} == "deploy_cert" ]; then
    echo " + Hook: Reloading Apache for ${2}..."
    service apache2 reload
    if [ ${2} == "campaignwiki.org" ]; then
	echo " + Hook: Regenerating prosody's files..."
	# 3 is privkey, 4 is cert, 5 is fullchain, 6 is chain, 7 is timestamp
	# use cat to preserve permissions and ownership
	for subdomain in chat auth.chat focus.chat jitsi-videobridge.chat conference.chat; do
	    cat ${5} > /etc/prosody/certs/$subdomain.campaignwiki.org.crt
	    cat ${3} > /etc/prosody/certs/$subdomain.campaignwiki.org.key
	done
	service prosody reload
    fi
    if [ ${2} == "alexschroeder.ch" ]; then
	echo " + Hook: Regenerating monit's .pem file..."
	# 3 is privkey, 4 is cert, 5 is fullchain, 6 is chain, 7 is timestamp
	# use cat to preserve permissions and ownership
	cat ${5} ${3} > /etc/ssl/localcerts/alexschroeder.ch.all.pem
	service monit reload
    fi
else
    echo " + Hook: Nothing to do..."
fi

Everything still seems to work. 🙂

– Alex Schroeder 2020-03-19 21:19 UTC


When I check the XMPP server with Bitlbee, however: “BitlBee does not support any of the offered SASL authentication schemes: ANONYMOUS” 😟

– Alex Schroeder 2020-03-19 21:21 UTC


Actually, now that I’ve talked to EionRobb on the #bitlbee channel, I think I understand.

  1. I need anonymous access to XMPP for Jitsi to work for anonymous users
  2. Internally, they get assigned random ids, but that doesn’t bother them because they access Jitsi via a URL that gives them a chat room and so whoever is there knew the URL and is your “friend” so you trust them and the alias they give themselves
  3. But from IRC via Bitlbee or via another XMPP client, that doesn’t make sense: you have an id first, like kensanata@pluspora.com, and then you send messages to other people whose id you trust

So, all of this wasn’t actually necessary! 😭

Not sure what to do now. Would a self-signed certificate for auth.chat.campaignwiki.org be enough?

Gah!

– Alex Schroeder 2020-03-19 21:44 UTC


OK, let’s figure out how this works. Prosody has uses this config file for the Jitsi stuff: /etc/prosody/conf.d/chat.campaignwiki.org.cfg.lua. From a copy in the global config file /etc/prosody/prosody.cfg.lua:

-- Certificates
-- Every virtual host and component needs a certificate so that clients and
-- servers can securely verify its identity. Prosody will automatically load
-- certificates/keys from the directory specified here.

OK, so which ones get listed?

  1. virtual server localhost (really?)
  2. virtual server chat.campaignwiki.org (with a SSL section pointing to /etc/prosody/certs)
  3. component conference.chat.campaignwiki.org (no SSL section)
  4. component jitsi-videobridge.chat.campaignwiki.org (no SSL section)
  5. virtual server auth.chat.campaignwiki.org (with a SSL section pointing to /etc/prosody/certs)
  6. component focus.chat.campaignwiki.org (no SSL section)

Hm. So I’m guessing only the virtual servers with SSL section actually need them. Instead of modifying the files too much, I will simply remove the unnecessary copies of the certificates and not add any self-signed certs back.


And for Monit:

check process prosody with pidfile /var/run/prosody/prosody.pid
    start program = "/bin/systemctl start prosody"
    stop program = "/bin/systemctl stop prosody"
    restart program = "/bin/systemctl restart prosody"
    if totalmem > 500 MB for 5 cycles then restart
    if 3 restarts within 15 cycles then stop

check process jicofo with pidfile /var/run/jicofo.pid
    start program = "/bin/systemctl start jicofo"
    stop program = "/bin/systemctl stop jicofo"
    restart program = "/bin/systemctl restart jicofo"
    if totalmem > 500 MB for 5 cycles then restart
    if 3 restarts within 15 cycles then stop

check process jitsi-videobridge with pidfile /var/run/jitsi-videobridge/jitsi-videobridge.pid
    start program = "/bin/systemctl start jitsi-videobridge"
    stop program = "/bin/systemctl stop jitsi-videobridge"
    restart program = "/bin/systemctl restart jitsi-videobridge"
    if totalmem > 500 MB for 5 cycles then restart
    if 3 restarts within 15 cycles then stop

– Alex Schroeder 2020-03-20 08:18 UTC


Had my first virtual tea via Jitsi with my wife in another room and a friend in the neighbourhood. Got to experiment with phone and laptop, camera and external microphone. 🙂

– Alex Schroeder 2020-03-20 15:02 UTC


I am trying to get Jitsi Meet to work. I have all the hostnames configured and a Let’s Encrypt cert for all of them, and I got nginx configured to show the web-interface. jitsi-videobridge is running and saying things in its logfile, but jicofo only prints a couple of warnings and then nothing more, which I think is wrong? (It is a Java-program after all). Everybody who connects can see video of themselves, and that others are present, but no video or audio from other participants. Any hints? What does jicofo log in your working setup?

Adam 2020-03-20 16:30 UTC


Ah, I got it working by adding the prosody user with the correct password and noticing that port 5347 is only listened on locally, so I should use localhost in jicofo/config and jitsi-videobridge/config. Thanks for the inspiration!

Adam 2020-03-20 17:04 UTC


Jicofo logs a ton of info messages and no errors: focus request, joining the room, chat room event, granted ownership, member joined, another focus request, another member joining the room, another chat room event, a conference created, a participant added, nothing out of the ordinary. There is one warning “no jingle session yet” immediately followed by an info log message saying a session-accept was received.

Back when the setup was not working, I had Java exceptions in there: java.lang.ClassNotFoundException, IllegalArgumentException, stuff like that.

Check for errors in your Prosody log files, too. /var/log/prosody/prosody.err had errors (”Error binding encrypted port for https: No certificate present in SSL/TLS configuration for https port 5281”) back when it wasn’t working (this was also visible in /var/log/prosody/prosody.log).

– Alex Schroeder 2020-03-20 17:05 UTC


Hah! Excellent. 🙂

I only have localhost listed in the Jicofo config file:

# grep HOST /etc/jitsi/*/config 
/etc/jitsi/jicofo/config:JICOFO_HOST=localhost
/etc/jitsi/jicofo/config:JICOFO_HOSTNAME=chat.campaignwiki.org
/etc/jitsi/videobridge/config:JVB_HOSTNAME=chat.campaignwiki.org
/etc/jitsi/videobridge/config:JVB_HOST=

– Alex Schroeder 2020-03-20 17:06 UTC


Thanks!

I have localhost in both HOST= lines; works.

I can’t make the Android app work, though - it just immediately says it was disconnected. I can’t see it trying to connect in the webserver nor the prosody log - did you do something special to get that working?

Adam 2020-03-20 18:39 UTC


Ah, the app didn’t work because I used the cert.pem instead of the fullchain.pem as the ssl-certificate in the webserver - now the Android app works as well!

Adam 2020-03-20 18:58 UTC


Perfect. Thanks for posting about your experience!

– Alex Schroeder 2020-03-20 23:29 UTC


Let’s see... Changing from “stable” to “unstable” in /etc/apt/sources.list.d/jitsi-stable.list and running apt update and apt list --upgradable:

jicofo/unstable 1.0-534-1 amd64 [upgradable from: 1.0-508-1]
jitsi-meet-prosody/unstable 1.0.3905-1 all [upgradable from: 1.0.3729-1]
jitsi-meet-web-config/unstable 1.0.3905-1 all [upgradable from: 1.0.3729-1]
jitsi-meet-web/unstable 1.0.3905-1 all [upgradable from: 1.0.3729-1]
jitsi-meet/unstable 1.0.4304-1 all [upgradable from: 1.0.4101-1]
jitsi-videobridge/unstable 1132-1 amd64 [upgradable from: 1126-1]

Unfortunately, however:

The following NEW packages will be installed:
  coturn jitsi-meet-turnserver libevent-core-2.1-6 libevent-extra-2.1-6 libevent-openssl-2.1-6 libevent-pthreads-2.1-6 libhiredis0.14
  libnginx-mod-http-auth-pam libnginx-mod-http-dav-ext libnginx-mod-http-echo libnginx-mod-http-geoip libnginx-mod-http-image-filter
  libnginx-mod-http-subs-filter libnginx-mod-http-upstream-fair libnginx-mod-http-xslt-filter libnginx-mod-mail libnginx-mod-stream nginx
  nginx-common nginx-full sqlite3 uuid-runtime
The following packages will be upgraded:
  jicofo jitsi-meet jitsi-meet-prosody jitsi-meet-web jitsi-meet-web-config jitsi-videobridge
6 upgraded, 22 newly installed, 0 to remove and 0 not upgraded.

I don’t really want nginx – I’m an Apache person!

– Alex Schroeder 2020-03-23 14:56 UTC


Installing Coturn to get rid of Google’s STUN servers.

– Alex Schroeder 2020-03-23 23:48 UTC


I think I’m running an anonymous TURN server for now. The things we do for Corona! 😇

apt install coturn

Edit /etc/default/coturn.conf and change it to TURNSERVER_ENABLED=1. Then systemctl restart coturn and systemctl restart jitsi-videobridge (I guess?). It’s still working, as far as I can tell.

– Alex Schroeder 2020-03-27 16:13 UTC


Also note 2020-03-26 Jitsi and the CPU.

– Alex Schroeder 2020-03-29 10:04 UTC

Add Comment

2020-02-16 Firefox syncserver

OK, so I’m trying to run my own syncserver. It’s separate from the account server, so you still need a Firefox Account. If you do things in the wrong order your browser syncs immediately – to their own sync server, which is not what I wanted. 😭

Anyway, let’s start from the beginning.

First, check out the project, syncserver. It uses Python 2. Oh well!

The Sync Server software runs using python 2.7, and the build process requires make and virtualenv.

I looked at this blog post by Matthias Dietel.

apt install python-dev git-core python-virtualenv g++

Easy enough! Build the project:

cd ~/src
git clone https://github.com/mozilla-services/syncserver.git
cd syncserver
make build
make test

Edit the syncserver.ini file. I’ve made the following changes:

diff --git a/syncserver.ini b/syncserver.ini
index ccf1ae0..21d0586 100644
--- a/syncserver.ini
+++ b/syncserver.ini
@@ -1,6 +1,6 @@
 [server:main]
 use = egg:gunicorn
-host = 0.0.0.0
+host = localhost
 port = 5000
 workers = 1
 timeout = 30
@@ -11,7 +11,7 @@ use = egg:syncserver
 [syncserver]
 # This must be edited to point to the public URL of your server,
 # i.e. the URL as seen by Firefox.
-public_url = http://localhost:5000/
+public_url = https://alexschroeder.ch/sync
 
 # By default, syncserver will accept identity assertions issued by
 # any BrowserID issuer.  The line below restricts it to accept assertions
@@ -20,7 +20,7 @@ public_url = http://localhost:5000/
 identity_provider = https://accounts.firefox.com/
 
 # This defines the database in which to store all server data.
-#sqluri = sqlite:////tmp/syncserver.db
+sqluri = sqlite:////home/alex/syncserver.db
 #sqluri = pymysql://sample_user:sample_password@127.0.0.1/syncstorage
 
 # This is a secret key used for signing authentication tokens.
@@ -34,7 +34,7 @@ identity_provider = https://accounts.firefox.com/
 
 # Set this to "false" to disable new-user signups on the server.
 # Only requests by existing accounts will be honoured.
-# allow_new_users = false
+allow_new_users = new
 
 # Set this to "true" to work around a mismatch between public_url and
 # the application URL as seen by python, which can happen in certain reverse-
@@ -55,4 +55,4 @@ force_wsgi_environ = false
 # MySQL based syncstorage-rs 1.5 server hosted at http://localhost:8000/1.5
 
 # "{node}/1.5/{uid}"
-# sync-1.5 = "http://localhost:8000/1.5/{uid}"
+sync-1.5 = "https://alexschroeder.ch/sync/token/{node}/sync/1.5/{uid}"

That is to say:

  1. the server only listens on localhost:5000
  2. the server can be reached at https://alexschroeder.ch/sync
  3. it uses a SQLite db at /home/alex/syncserver.db
  4. it allows new users to sync their stuff
  5. and I had to change that endpoint at the end... 🤷

So now I need to set up my website. I use Apache and I already have proxy stuff set up. All I had to add was this:

ProxyPass /sync	    http://localhost:5000/

Thus, any request starting with /sync to my existing website gets sent to port 5000 on localhost which happens to be the URL where the sync server is running.

Next, open about:config in the browser and change identity.sync.tokenserver.uri to https://alexschroeder.ch/sync/token/1.0/sync/1.5 – then I’m ready to open the browser preferences and sign in.

If everything worked, you should see your display name in the top spot of the hamburger menu. 🍔 😁

If you see a warning sign after a second or two, stuff isn’t working.

It all sounds so easy now, but it took me a while to get everything working, believe me. 😭

Now I’m seeing the following errors in the log output of make serve:

ERROR:syncserver:The public_url setting doesn't match the application url.
This will almost certainly cause authentication failures!
    public_url setting is: https://alexschroeder.ch/sync
    application url is:    http://localhost:5000/sync
You can disable this check by setting the force_wsgi_environ
option in your config file, but do so at your own risk.

I have no idea what it means. I’m thinking it works anyway? I’m making an additional change to syncserver.ini just to be sure.

force_wsgi_environ = true

The instructions tell me that we only need this in certain reverse-proxy setups but this is a reverse-proxy setup so I think I should be fine?

Now I wonder how to leave the sync server running... Perhaps it’s OK to just do this every now and then?

Now that I have it running I wonder about the utility of it all:

  1. there’s no need for me to mix my private Firefox and my work Firefox bookmarks and logins
  2. I currently cannot make Firefox on my iPad and my iPhone use a different sync address

Oh Apple, the golden prison, where I’m safe, but also where I’m locked in. Sigh. 😔

But wait… This issue is closed: Add ability to set custom identity.sync.tokenserver.uri for self-hosted Sync #5006?

One of the last comments on that issue, by user fireglow:

Just to chime in a little: Python 2.7, the flavor of Python the sync software is written in, will go End-of-Life at the end of this month, year, and decade. Mozilla already has indicated that there will be no rewrite for Python 3. I gather there’s a rewrite of these services in Rust in the works, at https://github.com/mozilla-services/syncstorage-rs It’s as of now unclear to me how all these parts will fit together in a way so us self-hosters will be able migrate over.

Oh wow.

I think I’m going to stop all of this. It’s making my head hurt. I checked the Rust rewrite project. Apparently you need to use either MySQL or Spanner as your DB. And with that, I’ve decided that not running this service and not syncing my stuff is probably for the best.

I undid all the changes, stopped the server, uninstalled the software, and deleted the Firefox account again.

Tags:

Comments on 2020-02-16 Firefox syncserver

Hi Alex,

You can set a custom sync server amd token server in Firefox ios.

To access it:

  • Enter the settings menu
  • Hit the version number 5 times, this opens debug mode
  • Under the sync option, there is a new menu item, Advanced Sync
  • The two settings are now available

Note that I haven’t done this, as I don’t (yet) have a sync server.

dgold 2020-02-16 19:25 UTC


I get the feeling that this doesn’t work on the iPad... I get a bunch of extra options but no place where I can set the sync server URL.

– Alex Schroeder 2020-02-16 19:28 UTC


I don’t know what to say to you. I set up my server in the way you described (except I didn’t set the last two options in the file)

I accessed the Advanced Settings in FFox on iPadOS. All works perfectly.

dgold 2020-02-16 20:24 UTC


How weird. I just updated Firefox on the iPad and it does look different, but I still don’t know where to start. Do you feel like pasting a screenshot, or can you explain it like I’m super confused? Because I feel like I am!

  1. open Firefox on the iPad
  2. click hamburger menu
  3. scroll down until you see “Version 22.0 (17157)
  4. tap this section five time until the list of options changes
  5. now what?

At the top, I see three items:

  1. my email address and a note saying I need to enter my password (but since I deleted my account, clicking on this item just takes me to accounts.firefox.com where I could probably recreated my account and start sync immediately but with their servers)
  2. error diagnosis: ask for an upgrade (translating from German)
  3. error diagnosis: forgot sync-status (translating from German)

No other item in these settings seem related to sync. What am I not seeing?

– Alex Schroeder 2020-02-16 22:08 UTC


After you tap the section until the lost of options changes, there should be a new menu item at the top of the Preferences under “Firefox Account” called “Advanced Sync Settings”.

In that modal dialog you can set the address of your sync server, in the form https://example.com/sync/token/1.0/sync/1.5

I didn’t create a FFox account until after I’d created my sync server.

dgold 2020-02-17 11:00 UTC


Oh wow, I just tried it on the iPhone and it worked. Tapped on the Version five times, it automatically scrolls to the top showing me the things I listed previously, but now I scroll back to the bottom and do it again and now I see new items at the top:

  1. Sign in to Sync
  2. Advanced Sync Settings
  3. Debug: use stage servers

Hah, thanks for pointing me at it again. I might give this another try.

– Alex Schroeder 2020-02-17 11:40 UTC


Well, I think it’s working:

Firefox on iOS Advanced Sync Settings Firefox on iOS Advanced Settings

On the iPad, the situation was different. I had to do the following:

  1. log into sync even though I couldn’t change the sync server and Firefox synced with the default sync server
  2. log out of sync
  3. activate the advanced settings again
  4. now you can change the sync server
  5. log into sync again

🙄

I’m looking at Run your own Sync-1.5 Server right now and I’m wondering about the WSGI setup. Are you using it?

I guess I don’t need to use it as my setup is basically Apache + Gunicorn. 😀

Except have to turn on force_wsgi_environ. As far as I can see in the Apache documentation on Reverse Proxy Request Headers but then again, the nginx setup instructions mention X-Forwarded-Proto and X-Real-IP, both of which the Apache docs don’t mention. Gaaaaah! 😭

Setup for Monit:

check process syncserver matching syncserver
    start program = "/bin/bash -c '(cd /home/alex/src/syncserver && exec /usr/bin/make serve 2&>1 syncserver.log)' &"
    stop program = "/usr/bin/killall gunicorn"
    if failed
	host alexschroeder.ch
	port 443
	type tcpssl
	protocol http
	request "/sync"
	for 5 cycles
	then restart
    if totalmem > 100 MB for 5 cycles then restart
    if 3 restarts within 15 cycles then stop

– Alex Schroeder 2020-02-17 21:19 UTC

Add Comment

2020-01-31 Banning myself with fail2ban

Recently I have noticed that I’m sometimes banned from my own websites. That is, the site is not reachable, but when I check any of the “is the site down for everybody or is it just me?” sites, it’s always just me. I also cannot SSH to the machine unless I use the IPv6 address directly.

OK, so fail2ban is banning the IPv4 of my home network. Why? Is some app I’m using bombarding the site with requests? Let’s check.

Visiting ip4.me tells me my IPv4. Grepping /var/log/fail2ban.log I see that it has been banned at 11:03 and 13:03 today, by the alex-apache rule.

This rules counts every access as a potential fail and ignores some URLs that I deem to be harmless (static files such as CSS files, fonts, pictures, podcast episodes, PDF files, and so on:

failregex = ^(www\.)?(alexschroeder\.ch|arabisch-lernen\.org|campaignwiki\.org|communitywiki\.org|emacswiki\.org|flying-carpet\.ch|korero\.org|oddmuse\.org|orientalisch\.info):[0-9]+ <HOST>

ignoreregex = ^[^"]*"GET /(robots\.txt |favicon\.ico |[^/ ]+.(css|js) |cgit/|css/|fonts/|pics/|export/|podcast/|1pdc/|gallery/|static/|munin/|osr/|indie/|rpg/|face/|traveller/|hex-describe/|text-mapper/|contrib/pics/)

This is from /etc/fail2ban/filter.d/alex-apache.conf. The actual limits are defined in /etc/fail2ban/jail.d/alex.conf:

[alex-apache]
enabled = true
port    = http,https
logpath = %(apache_access_log)s
findtime = 40
maxretry = 20

So basically you’re allowed 20 requests in 40s, not counting the requests matching ignoreregex.

OK, let’s see the requests. Here’s a little Perl script I wrote:

#!/usr/bin/env perl
while (<STDIN>) {
  m/^(\S+:\d+) ([0-9.]+) - - \[(.*?)\] "(.*?)" (\d+) (\d+|-) "(.*?)" "(.*?)"/ or warn "Cannot parse:\n$_" and next;
  my ($host, $ip4, $date, $request, $code, $size, $referrer, $agent) = ($1, $2, $3, $4, $5, $6, $7, $8);
  $requests{$request}++;
  $total++;
}
@result = sort {$requests{$b} <=> $requests{$a}} keys %requests;
foreach $label (@result) {
  printf "%70s %10d   %3d%%\n", $label, $requests{$label}, 100* $requests{$label} / $total;
}

Let’s use it to report on today’s log entries from my IP:

    GET /pdfs/spellcasters/ HTTP/1.1         12    19%
                 GET /pdfs/ HTTP/1.1         12    19%
               GET /jewelry HTTP/1.1          8    13%
             GET /wiki/Apps HTTP/1.1          8    13%

Hm, who the hell is requesting /pdfs/spellcasters/ and /pdfs/‽ Not me, that’s for sure! At least not that I know of. Then again, I’d say that the /pdfs directory contains just static files so adding that to the ignoreregex should be no problem.

I still wonder who does this, though. Is it the Firefox app on iOS?

Tags:

Add Comment

2019-12-19 Oddmuse 6 memory use

Process RSS summed

I have a Munin module that runs ps -eo rss,command | grep $prog for various regular expressions.

Well, actually I sum it all up so I run this:

ps -eo rss,command | gawk '
BEGIN              { total = "U"; } # U = Unknown.
/grep/             { next; }
'"/perl6/"'        { total = total + $1; }
END                { print total; }'

For Oddmuse 6 running on Cro and Perl 6 the result is 385120, i.e. 385 MB. That doesn’t quite match the data in the image (about 350MB) but it’s close enough. Compare that to the regular Oddmuse instances running on Perl 5 with Mojolicious...

Am I comparing apples to oranges? Let’s see the ps output for this stuff and compare it to the output of a regular Oddmuse wiki:

alex@sibirocobombus:~$ ps -eo rss,command | grep perl6
  892 grep perl6
358120 /home/alex/rakudo/bin/moar --execname=/home/alex/rakudo/bin/perl6 --libpath=/home/alex/rakudo/share/nqp/lib --libpath=/home/alex/rakudo/share/nqp/lib --libpath=/home/alex/rakudo/share/perl6/lib --libpath=/home/alex/rakudo/share/perl6/runtime /home/alex/rakudo/share/perl6/runtime/perl6.moarvm service.p6
alex@sibirocobombus:~$ ps -eo rss,command | grep campaignwiki
 5104 /home/alex/campaignwiki.org/gridmapper-server.pl
  892 grep campaignwiki
32924 /home/alex/farm/campaignwiki2.pl
 7784 /home/alex/farm/campaignwiki2.pl
32460 /home/alex/farm/campaignwiki2.pl
17712 /home/alex/campaignwiki.org/gridmapper-server.pl
alex@sibirocobombus:~$ ps -eo rss,command | gawk '
BEGIN              { total = "U"; } # U = Unknown.
/grep/             { next; }
'"/campaignwiki2/"'        { total = total + $1; }
END                { print total; }'
73168

So, 358120 for Oddmuse 6 looks correct, and the 73168 for Campaign Wiki looks correct. Both act as web servers behind Apache, both run a wiki.

Tags:

Add Comment

2019-11-16 Pleroma

OK, so I tried Epicyon and it used a lot of CPU cycles doing not much. Would Pleroma fare any better?

I am ready to give it a try: Installing on Debian Based Distributions.

apt install postgresql postgresql-contrib elixir erlang erlang-dev erlang-tools erlang-parsetools erlang-ssh erlang-xmerl

Well, I was following the instructions until I got to this step:

root@sibirocobombus:/opt/pleroma# sudo -Hu pleroma mix deps.get
!!! RUNNING IN LOCALHOST DEV MODE! !!!
FEDERATION WON'T WORK UNTIL YOU CONFIGURE A dev.secret.exs
Could not find Hex, which is needed to build dependency :phoenix
Shall I install Hex? (if running non-interactively, use "mix local.hex --force") [Yn] 
** (UndefinedFunctionError) function :inets.stop/2 is undefined (module :inets is not available)
    :inets.stop(:httpc, :mix)
    (mix) lib/mix/utils.ex:560: Mix.Utils.read_httpc/1
    (mix) lib/mix/utils.ex:501: Mix.Utils.read_path/2
    (mix) lib/mix/local.ex:149: Mix.Local.read_path!/2
    (mix) lib/mix/local.ex:126: Mix.Local.find_matching_versions_from_signed_csv!/2
    (mix) lib/mix/tasks/local.hex.ex:56: Mix.Tasks.Local.Hex.run_install/1
    (mix) lib/mix/dep/loader.ex:168: Mix.Dep.Loader.with_scm_and_app/4
    (mix) lib/mix/dep/loader.ex:121: Mix.Dep.Loader.to_dep/3

This helped:

apt install erlang-inets

🙂

I also changed registrations_open to false in config/prod.secret.exs.

Tags:

Comments on 2019-11-16 Pleroma

And I wrote my first Elixir code with the help of @wim_v12e:

diff --git a/lib/mix/tasks/pleroma/user.ex b/lib/mix/tasks/pleroma/user.ex
index a3f8bc945..964209c96 100644
--- a/lib/mix/tasks/pleroma/user.ex
+++ b/lib/mix/tasks/pleroma/user.ex
@@ -442,6 +442,20 @@ defmodule Mix.Tasks.Pleroma.User do
     end
   end
 
+  def run(["list"]) do
+    start_pleroma()
+
+    Pleroma.User.Query.build(%{local: true})
+    |> Pleroma.RepoStreamer.chunk_stream(500)
+    |> Stream.each(fn users ->
+      users
+      |> Enum.each(fn user ->
+        shell_info("#{user.nickname} moderator: #{user.info.is_moderator}, admin: #{user.info.is_admin}, locked: #{user.info.locked}, deactivated: #{user.info.deactivated}")
+      end)
+    end)
+    |> Stream.run()
+  end
+
   defp set_moderator(user, value) do
     info_cng = User.Info.admin_api_update(user.info, %{is_moderator: value})

This allows me to run:

root@sibirocobombus:/opt/pleroma# sudo -Hu pleroma MIX_ENV=prod mix pleroma.user list
Compiling 1 file (.ex)
internal.fetch moderator: false, admin: false, locked: false, deactivated: false
alex moderator: false, admin: true, locked: false, deactivated: false
admin moderator: false, admin: true, locked: false, deactivated: false
kensanata moderator: false, admin: false, locked: false, deactivated: false

– Alex Schroeder 2019-11-17 13:29 UTC


For now I’m happy with Pleroma. Notice how load came back down again after switching from Epicyon to Pleroma and PostgreSQL. I wasn’t running a database before the Pleroma installation.

Load

– Alex Schroeder 2019-11-17 21:11 UTC


It’s still running on my server without acting up. I’m happy even though I had to install a database for it. Should I ever want a lightweight server, another project to look at would be Kibou. It’s written in Rust instead of Erlang+Elixir. Not sure whether I like that better but there it is.

– Alex Schroeder 2019-12-10 21:58 UTC


Upgrade:

sudo -Hu pleroma git pull
sudo -Hu pleroma mix deps.get
sudo systemctl stop pleroma
sudo -Hu pleroma MIX_ENV=prod mix ecto.migrate
sudo systemctl start pleroma

– Alex Schroeder 2020-01-02 14:23 UTC


Actually, now that I think about, I think I don’t really use it for anything. I’ll simply uninstall it again. It worked. It was nice. Time to do something else.

– Alex Schroeder 2020-01-07 21:39 UTC

Add Comment

2019-11-09 Upgrading Debian from Stretch to Buster

I did it! I upgrade the server. No restart required. I was a bit intimidated by the extensive documentation. But I was fine.

First, I improved my backup. I currently use the following:

#!/bin/bash

# Using sudo rsync --archive to preserve ownership.
# Using --fake-super to avoid changes to groups and owners

echo Backing up Sibirocobombus
echo home directory
rsync --archive --fake-super --verbose --compress --delete --delete-excluded \
      --itemize-changes \
      --exclude '/home/alex/logs' \
      --exclude '/home/alex/alexschroeder.ch/share' \
      --exclude '/home/alex/planet/osr/' \
      --exclude '/home/alex/planet/indie/' \
      --exclude '.cpan/build' \
      --exclude '.cpan/sources' \
      --exclude '.cpanplus' \
      --exclude '.cpanm' \
      --exclude '.cache' \
      --exclude '.Trash' \
      --exclude '.local/share/Trash' \
      --exclude 'temp/' \
      --exclude 'pids/' \
      --exclude 'visitors.log' \
      --exclude 'referer/' \
      --exclude '.git/' \
      --rsh="ssh -p 882" \
      root@alexschroeder.ch:/home \
      /home/alex/Documents/Sibirocobombus

# https://www.debian.org/releases/buster/amd64/release-notes/ch-upgrading.en.html#data-backup
echo etc directory
rsync --archive --fake-super --verbose --compress --delete --delete-excluded \
      --itemize-changes \
      --rsh="ssh -p 882" \
      root@alexschroeder.ch:/etc \
      root@alexschroeder.ch:/var/lib/dpkg \
      root@alexschroeder.ch:/var/lib/apt/extended_states \
      root@alexschroeder.ch:/var/lib/dehydrated \
      root@alexschroeder.ch:/usr/lib/cgit/filters \
      /home/alex/Documents/Sibirocobombus

I also installed and ran apt-forktracer:

apt install apt-forktracer
apt-forktracer | sort > packages-not-from-debian

I also checked my system using dpkg --audit.

I ran everything inside script -t 2>~/upgrade-busterstep.time -a ~/upgrade-busterstep.script.

I was very afraid!

But eventually I just changed my /etc/apt/sources.list to the following:

deb http://deb.debian.org/debian buster main non-free contrib
deb http://security.debian.org/debian-security buster/updates main contrib non-free

# deb  http://ftp.debian.org/debian stretch main non-free contrib
# deb-src  http://ftp.debian.org/debian stretch main non-free contrib

# deb  http://ftp.debian.org/debian stretch-updates main non-free contrib
# deb-src  http://ftp.debian.org/debian stretch-updates main non-free contrib

# deb http://security.debian.org/ stretch/updates main non-free contrib
# deb-src http://security.debian.org/ stretch/updates main non-free contrib

I guess I should think about updates and security? Or perhaps they aren’t used anymore? I have no idea. I’ll look at this some other time.

Finally I ran the upgrade itself:

apt update
apt upgrade
apt dist-upgrade

I had two changes to system config files:

  1. in /etc/logrotate.conf I changed weeklydaily (keeping 4 days worth of logs instead of 4 weeks)
  2. in /etc/ssh/sshd_config I merged my changes with what upstream had provided

I am amazed to see that the server is still up. It didn’t even reboot. Let me sing the praise of Debian and all the things it includes and all the volunteers and maintainers and testers—thank you all. ❤️

I think I’m going to be fine. 😃

I noticed that I had to reinstall all my pip3 stuff.

For the Mastodon bots:

pip3 install Mastodon.py cairosvg html2text

For Epicyon:

pip3 install requests commentjson beautifulsoup4 pycryptodome

My monitoring also broke down.

Monit stopped monitoring a bunch of stuff because the checksums failed.

Munin isn’t running at all...

Well, time to look at this tomorrow!

Tags:

Comments on 2019-11-09 Upgrading Debian from Stretch to Buster

Cleaning up non-default config files... I installed debsums and use debsums -ce to find config files that I have changed.

Then I use a little script I got from StackExchange to find the diff.

#!/bin/bash

# Usage: debdiffconf.sh FILE
# Produce on stdout diff of FILE against the first installed Debian package
# found that provides it.
# Returns the exit code of diff if everything worked, 3 or 4 otherwise.

command -v apt-get >/dev/null 2>&1 || {
  echo "apt-get not found, this is probably not a Debian system. Aborting." >&2;
  exit 4; }
command -v apt-file >/dev/null 2>&1 || {
  echo "Please install apt-file: sudo apt-get install apt-file. Aborting." >&2;
  exit 4; }

FILE=$(readlink -f "$1")
while read PACKAGE
do
  # verify from first installed package
  if dpkg-query -W --showformat='${Status}\n' | grep installed > /dev/null
  then
    DIR=$(mktemp -d)
    cd "$DIR"
    echo "Trying $PACKAGE..." >&2
    apt-get download "$PACKAGE" >&2
    # downloaded archive is the only file present...
    ARCHIVE=$(ls)
    mkdir contents
    # extract entire archive
    dpkg-deb -x "$ARCHIVE" contents/ >&2
    if [ -f "contents$FILE" ]
    then
      # package contained required file
      diff "contents$FILE" "$FILE"
      RET=$?
      # cleanup
      cd
      rm -Rf "$DIR"
      # exit entire script as this is the main shell
      # with the return code from diff
      exit $RET
    else
      # cleanup
      cd
      rm -Rf "$DIR"
    fi
  fi
done < <(apt-file -l search "$FILE")
# if we are here, it means we have found no suitable package
echo "Could not find original package for $FILE" >&2
exit 3

And if I think the changes I made could be reverted, I run patch --reverse $FILE on the file and paste the diff seen above to revert it.

– Alex Schroeder 2019-11-10 11:38 UTC


The problem I had seems to due to changes I made to the Apache configuration. Apparently I created a special account with password to allow access to Munin from remote systems. This makes sense since I’m running Munin on the server...

*** apache24.conf~	2019-05-16 01:21:08.000000000 +0200
--- apache24.conf	2019-11-10 12:47:00.533056622 +0100
***************
*** 14,25 ****
  Alias /munin/static/ /var/cache/munin/www/static/
  
  <Directory /var/cache/munin/www>
!     Require local
      Options None
  </Directory>
  
  <Directory /usr/lib/munin/cgi>
!     Require local
      <IfModule mod_fcgid.c>
          SetHandler fcgid-script
      </IfModule>
--- 14,36 ----
  Alias /munin/static/ /var/cache/munin/www/static/
  
  <Directory /var/cache/munin/www>
!     Order allow,deny
!     Allow from all
      Options None
+     AuthUserFile /etc/munin/munin-htpasswd
+     AuthName "Munin"
+     AuthType Basic
+     require valid-user
  </Directory>
  
  <Directory /usr/lib/munin/cgi>
!     Order allow,deny
!     Allow from all
!     Options None
!     AuthUserFile /etc/munin/munin-htpasswd
!     AuthName "Munin"
!     AuthType Basic
!     require valid-user
      <IfModule mod_fcgid.c>
          SetHandler fcgid-script
      </IfModule>

– Alex Schroeder 2019-11-10 11:51 UTC

Add Comment

2019-11-09 Epicyon needs Python 3.6 which needs a Debian upgrade

I’ve tried to install Epicyon on my server but I’m running into a Python problem. I think I have everything set up correctly (my notes below) but I’m getting a syntax error in webfinger.py:

python3 webfinger.py 
  File "webfinger.py", line 81
    return f"data:application/magic-public-key,RSA.{mod}.{pubexp}"
                                                                 ^
SyntaxError: invalid syntax

I’m using Debian stable (9.11) on my server which means I have Python 3.5.3 installed. f-strings joined Python in 3.6, apparently.

So... Upgrade my server? Make Epicyon work with Python 3.5? Postpone the entire thing?

I don’t know...

Epicyon

I was more or less following the official installation instructions.

Some decisions:

  • I’m not using certbot because I am using dehydrated.
  • I don’t want to install in /etc/epicyon. Let me use /home/epicyon instead.

Install the code as root:

apt-get -y install tor python3-pip python3-socks imagemagick python3-numpy python3-setuptools python3-crypto python3-dateutil python3-pil.imagetk
pip3 install commentjson beautifulsoup4 pycryptodome
adduser --system --home=/home/epicyon --group epicyon
cd /home
git clone https://gitlab.com/bashrc2/epicyon
cd epicyon
./theme purple
chown -R epicyon:epicyon /home/epicyon

Create the demon by creating the file /etc/systemd/system/epicyon.service as follows:

[Unit]
Description=epicyon
After=syslog.target
After=network.target

[Service]
Type=simple
User=epicyon
Group=epicyon
WorkingDirectory=/home/epicyon
ExecStart=/usr/bin/python3 /home/epicyon/epicyon.py --port 443 --proxy 7156 --domain fedi.alexschroeder.ch --registration open --debug
Environment=USER=epicyon
Restart=always
StandardError=syslog
CPUQuota=30%

[Install]
WantedBy=multi-user.target

Activate the demon:

systemctl enable epicyon
systemctl start epicyon

Check how it is doing:

journalctl -u epicyon

On my first attempt I had forgotten to install the Python modules using pip3 and saw the error messages in the journal alerting me to the fact.

As for the website, I’m using Apache. I’m on my own, now. I edited /etc/apache2/sites-enabled/100-alexschroeder.ch.conf because that’s where my site is configured and added the following:

  • a redirect from HTTP on port 80 to HTTPS on port 443
  • the right server name
  • SSL files generated by dehydrated
  • proxying to port 7156

I’m basically trusting my remaining SSL and security settings.

Here’s what I added to the site configuration:

<VirtualHost *:80>
    ServerName fedi.alexschroeder.ch
    Redirect permanent / https://fedi.alexschroeder.ch/
</VirtualHost>
<VirtualHost *:443>
    ServerName fedi.alexschroeder.ch
    SSLEngine on
    SSLCertificateFile      /var/lib/dehydrated/certs/alexschroeder.ch/cert.pem
    SSLCertificateKeyFile   /var/lib/dehydrated/certs/alexschroeder.ch/privkey.pem
    SSLCertificateChainFile /var/lib/dehydrated/certs/alexschroeder.ch/chain.pem
    SSLVerifyClient None
    ProxyPass /             http://alexschroeder.ch:7156/
</VirtualHost>

Next I had to tell my domain name service provider (Gandi) about the new subdomain. In the DNS zone file, I had to add a line for IPv4 and IPv6:

fedi 10800 IN A 178.209.50.237
fedi 10800 IN AAAA 2a02:418:6a04:178:209:50:237:1

Then I had to tell dehydrated about the new subdomain by editing /etc/dehydrated/domains.txt and changing the line for my site to:

alexschroeder.ch www.alexschroeder.ch fedi.alexschroeder.ch

With that, I was ready to regenerate the certificates (still doing everything as root). The version of dehydrated in Debian isn’t working for me so I have to use a copy of the master branch I checked out.

/home/alex/src/dehydrated/dehydrated -c

And now I should be ready!

  • the domain name server points to the right machine
  • on that machine, the web server knows how to redirect from HTTP to HTTPS for this domain
  • the web server also knows how to proxy HTTPS to port 7156 locally
  • the application is running and listening on the port

Or is it? I’m getting an error! journalctl -u epicyon shows the following:

Nov 09 14:56:01 sibirocobombus python3[27300]: Traceback (most recent call last):
Nov 09 14:56:01 sibirocobombus python3[27300]:   File "/home/epicyon/epicyon.py", line 9, in <module>
Nov 09 14:56:01 sibirocobombus python3[27300]:     from person import createPerson
Nov 09 14:56:01 sibirocobombus python3[27300]:   File "/home/epicyon/person.py", line 19, in <module>
Nov 09 14:56:01 sibirocobombus python3[27300]:     from webfinger import createWebfingerEndpoint
Nov 09 14:56:01 sibirocobombus python3[27300]:   File "/home/epicyon/webfinger.py", line 81
Nov 09 14:56:01 sibirocobombus python3[27300]:     return f"data:application/magic-public-key,RSA.{mod}.{pubexp}"
Nov 09 14:56:01 sibirocobombus python3[27300]:                                                                  ^
Nov 09 14:56:01 sibirocobombus python3[27300]: SyntaxError: invalid syntax

Tags:

Comments on 2019-11-09 Epicyon needs Python 3.6 which needs a Debian upgrade

I decided to stop Epicyon for now. It was basically using up all the CPU resources I had granted it via systemd. With no actual activity, I find that unacceptable.

Load goes up

Sometime between November 10 and 11 I managed to get it up and running and load goes up, and stays up.

– Alex Schroeder 2019-11-14 21:30 UTC


A lot of back and forth on this issue. Increase the CPU quota? Work on debugging this? What’s the problem?

Or should I just try Pleroma instead?

– Alex Schroeder 2019-11-16 22:09


I think I’ll give Pleroma a try...

systemctl stop epicyon
systemctl disable epicyon
apt remove tor python3-socks python3-numpy python3-setuptools \
  python3-crypto python3-dateutil python3-pil.imagetk

– Alex Schroeder 2019-11-17 11:09

Add Comment

2019-09-25 RAM doubled on this VM

A few days ago I received an email: if I power cycled the VM all my sites are running on, I’d get my RAM doubled for no reason. Ohhhh, cool! But...

Power cycling my server always scares me – but they promised to double my RAM if I did it… What to do! What to do? What to do‽ 😨

I did it.

Monit shows all the processes restarting

Well, actually I did a soft reset first and I think that didn’t do it! They really want me to power cycle the thing. So here we go, again! 😱

Or did it? Anyway, I did it again, but this time I really did a power toggle.

And it seems to have worked.

Munin shows RAM going from 3GB to 6GB

I don't understand all the lines on that graph, to be honest. What’s that green “committed” line doing, for example? On ServerFault somebody wrote:

Committed memory is, essentially, all the memory which has been allocated by applications, whether it’s used or not. In contrast, the “apps” is memory that is allocated AND used.

OK! Lots of unused memory allocated to apps... Looks like I have a ton of dead apps hanging around or something?

The price we pay is always interesting, of course. I’m paying 15€/month for this kernel-based virtual machine (KVM). And apparently that now gets me one core at 2.3GHz, 6GB RAM and around 75GB of disk space.

alex@sibirocobombus:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            3.0G     0  3.0G   0% /dev
tmpfs           598M  7.9M  590M   2% /run
/dev/vda1        73G   32G   39G  46% /
tmpfs           3.0G     0  3.0G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           3.0G     0  3.0G   0% /sys/fs/cgroup
tmpfs           598M     0  598M   0% /run/user/1000

No idea how that compares to others.

Tags:

Add Comment

More...

Comments


Please make sure you contribute only your own work, or work licensed under the GNU Free Documentation License. Note: in order to facilitate peer review and fight vandalism, we will store your IP number for a number of days. See Privacy Policy for more information. See Info for text formatting rules. You can edit the comment page if you need to fix typos. You can subscribe to new comments by email without leaving a comment.

To save this page you must answer this question:

Please say HELLO.