XMPP Server - Prosody

[djxx]

@nebulon understood. If needed, I'm sure we could make it not crash, even if it's not really a functional XMPP server. And for me I'm at least happy I have a way to run XMPP on my server. I'm just trying to share the work with others.

When is the next version scheduled for release?

[nebulon]

We are currently quite busy with Cloudron 9 development, lets see if we can get some of those things still in.

After reading up a bit on prosody docs, I still dont fully understand what we should do with the default location and what we could serve up as http (currently it shows a 404 error page for example). I saw there is also BOSH which is intended for http, so maybe that can be placed there instead, but so far I don't have a good grip on prosody's use of domains, certs and ports.

[djxx]

@nebulon In general, XMPP isn't going to actually serve any web pages. I installed an extra module for health checking because Cloudron requires it - but there's no reason it needs to be exposed externally. BOSH may use HTTP, but it doesn't serve responses that can be interpreted as web pages.

Personally, I don't think it hurts that there's a 404 on an HTTP/S port for an application that doesn't serve HTTP. Putting up a big page like "You found an XMPP server!" just invites trouble from the bad people on the internet.

[nebulon]

Thanks for that input. I think what we really need to add is a way to run apps as headless services. We already have a few of those like game servers and synapse (matrix) which have to serve up a placeholder page. That is not ideal indeed. After Cloudron 9 we will also look into other things like maybe databases, they would have very similar requirements then.

Sorry that this is not yet implemented but we will get there in the future, so it is already great to collect requirements for such service type apps like prosody. This helps to avoid developing the wrong things on the platform side

[robi]

Doesn't Matrix also require something on the root domain? Isn't that where the .well-known endpoint lives?

Something similar can be had?

@djxx, was the architecture chosen to use subdomains vs different ports?

If we're clever with the reverse proxy or with the main app server that listens to connections, it can ferry connections for other services by the request type or layer it needs to access, avoiding the need for extra ports or subdomains.

As for the HTTP page, why not make it multi-useful. For example, by default it's just a Cloudron OIDC login button and then per app it's configured what it does.. for backend services it could be a simple CLI stats page in HTML, or even a web terminal. That way it's flexible and useful to the admin, users, packaging and platform maintainers.

What else would be useful there?

[djxx]

@robi I'm not an expert, but I did do some research on this to respond to Nebulon above. The main thing to note here is that the XMPP protocol expects different domains for different functions. So yes - it's an architectural choice that is mostly beyond our control. Each subdomain represents a different module, and the modules talk to each other (and other servers). They need a way to identify themselves, and they use the domain.

I did some reading and it seems it is theoretically possible to make a server that utilizes different ports to differentiate the modules under a single sub-domain, but this would require tweaking and re-compiling an XMPP server.

Even if this was done, and all XMPP protocols followed, there's always the chance that some not-fully-compliant client that works with every well known XMPP server would not work with ours because we've chosen to deviate so far from the norm, while still technically following the standards. Making these upstream changes would require someone much more familiar with XMPP servers than me.

While I look forward to Cloudron allowing us to package more complex services, I think XMPP is in a pretty good place for the above-average admin; it currently takes less than 5 minutes to set it up. I will probably end up making a cron job to sync the certs into my app's storage volume when I deploy this to production. If I think it's useful to others, and Cloudron 9 isn't out yet, I'll share it here.

[robi]

@djxx Appreciate the response.

We already have some apps that require a secondary domain on the front end, but that isn't necessarily the case for the backend. We just need Nginx to route things as expected by the clients, etc.

Glad you're here, packaging and interested in this stuff working.

[djxx]

@nebulon - as you may see from my other posts, I'm all in on Cloudron ;). I'm in the process of migrating my last server from NethServer to Cloudron; XMPP is one of those services I couldn't move without. I plan to move forward with my manual approach in this thread, but am still interested to see when this could become an official app. Does Cloudron 9 have an ETA?

[nebulon]

@djxx we currently have no ETA for Cloudron 9, but also for this specific case, we haven't implemented anything yet.

I do think eventually we will get there to have better support for non-ui or just backend apps so to speak.

[djxx]

I'm happy to say that I've moved my XMPP server from NethServer to Cloudron. While this is probably not a common move, I am sharing some notes here in case it helps someone else. Also, perhaps this'll cause Cloudron to show up in a few more searches.

• Install XMPP on Cloudron using the steps above. A bit manual for now!

• Dump your ejabberd data (that's the XMPP server NethServer uses) with this command:
  /opt/ejabberd-20.04/bin/ejabberdctl --config-dir /etc/ejabberd dump /etc/ejabberd/xmpp_dump.txt

• Download this dump file locally

• For ease, clone the source for prosody to your local computer so you can utilize the migration tools and not install needless packages on Cloudron. You'll need to run ./configure and ./make - but you don't need to actually install it.

• Don't be a Lua noob. I spent a while struggling to get my Lua environment setup, and thought I needed to run the tools like lua ejabberd2prosody.lua but got lots of errors about dependencies missing. Once I figured out you need to execute it directly like ./ejabberd2prosody.lua things worked fine.

• run the ejabberd2prosody.lua script on your xmpp_dump.txt file:
  ./tools/ejabberd2prosody.lua ~/Desktop/xmpp_migrate/xmpp_dump.txt

• Create a migrator configuration (or use the one I've pasted below). It basically takes everything from the file data format and puts it into the sqlite format, since that's how the Cloudron prosody is configured. Docs:

  • https://prosody.im/doc/migrator
  • https://prosody.im/doc/storage
     

• Run the migrator script:
  ./tools/migration/prosody-migrator.lua --config=./migrator.cfg.lua prosody_files database

• Turn off your Cloudron XMPP app

• Copy the resulting prosody.sqlite file into your Cloudron XMPP's /app/data folder. It will be in the /data folder under your local prosody directory.

• Turn on your Cloudron XMPP app

Your bookmarks, rosters, etc. will now be transferred to your new server! This doesn't appear to move archive messages (mod_mam). Probably because most prosody servers aren't configured to store these permanently so they don't bother migrating them.

I only noticed one issue while migrating. When I first ran the migrator script it gave me errors about topics being empty on some MUCs. I thought I was being smart and edited the code to handle the blanks. This caused me to be unable to join the MUCs on Prosody on certain XMPP clients because Prosody expects there to be a Topic for every MUC.

Once I manually adjusted the MUC topics to be non-empty, the other clients started working fine.

Another almost-issue is that Gajim needed to be restarted a few times to start using OMEMO properly. I think the other MUC issues may have thrown it into an error state.

prosody_files {
    hosts = {
        -- each VirtualHost to be migrated must be represented
        ["domain.com"] = {
            "accounts";
            "account_details";
            "account_flags";
            "account_roles";
            "accounts_cleanup";
            "auth_tokens";
            "invite_token";
            "roster";
            "vcard";
            "vcard_muc";
            "private";
            "blocklist";
            "privacy";
            "archive";
            "archive_cleanup";
            "archive_prefs";
            "muc_log";
            "muc_log_cleanup";
            "persistent";
            "config";
            "state";
            "cloud_notify";
            "cron";
            "offline";
            "pubsub_nodes";
            "pubsub_data";
            "pep";
            "pep_data";
            "skeletons";
            "smacks_h";
            "tombstones";
            "upload_stats";
            "uploads";
        };
        ["conference.domain.com"] = {
            "accounts";
            "account_details";
            "account_flags";
            "account_roles";
            "accounts_cleanup";
            "auth_tokens";
            "invite_token";
            "roster";
            "vcard";
            "vcard_muc";
            "private";
            "blocklist";
            "privacy";
            "archive";
            "archive_cleanup";
            "archive_prefs";
            "muc_log";
            "muc_log_cleanup";
            "persistent";
            "config";
            "state";
            "cloud_notify";
            "cron";
            "offline";
            "pubsub_nodes";
            "pubsub_data";
            "pep";
            "pep_data";
            "skeletons";
            "smacks_h";
            "tombstones";
            "upload_stats";
            "uploads";
        };
    };

    type = "internal"; -- the default file based backend
    path = "/home/user/code/prosody-build/prosody-0.12.4/data/";
}

database {
    -- The migration target does not need 'hosts'
    type = "sql";
    driver = "SQLite3";
    database = "prosody.sqlite";
}