Docker seems to be the new official way of installing Prosody.

• Install XMPP on Cloudron using the steps above. A bit manual for now!

• Dump your ejabberd data (that's the XMPP server NethServer uses) with this command:
  /opt/ejabberd-20.04/bin/ejabberdctl --config-dir /etc/ejabberd dump /etc/ejabberd/xmpp_dump.txt

• Download this dump file locally

• For ease, clone the source for prosody to your local computer so you can utilize the migration tools and not install needless packages on Cloudron. You'll need to run ./configure and ./make - but you don't need to actually install it.

• Don't be a Lua noob. I spent a while struggling to get my Lua environment setup, and thought I needed to run the tools like lua ejabberd2prosody.lua but got lots of errors about dependencies missing. Once I figured out you need to execute it directly like ./ejabberd2prosody.lua things worked fine.

• run the ejabberd2prosody.lua script on your xmpp_dump.txt file:
  ./tools/ejabberd2prosody.lua ~/Desktop/xmpp_migrate/xmpp_dump.txt

• Create a migrator configuration (or use the one I've pasted below). It basically takes everything from the file data format and puts it into the sqlite format, since that's how the Cloudron prosody is configured. Docs:

  • https://prosody.im/doc/migrator
  • https://prosody.im/doc/storage
     

• Run the migrator script:
  ./tools/migration/prosody-migrator.lua --config=./migrator.cfg.lua prosody_files database

• Turn off your Cloudron XMPP app

• Copy the resulting prosody.sqlite file into your Cloudron XMPP's /app/data folder. It will be in the /data folder under your local prosody directory.

• Turn on your Cloudron XMPP app

Your bookmarks, rosters, etc. will now be transferred to your new server! This doesn't appear to move archive messages (mod_mam). Probably because most prosody servers aren't configured to store these permanently so they don't bother migrating them.

I only noticed one issue while migrating. When I first ran the migrator script it gave me errors about topics being empty on some MUCs. I thought I was being smart and edited the code to handle the blanks. This caused me to be unable to join the MUCs on Prosody on certain XMPP clients because Prosody expects there to be a Topic for every MUC.

Once I manually adjusted the MUC topics to be non-empty, the other clients started working fine.

Another almost-issue is that Gajim needed to be restarted a few times to start using OMEMO properly. I think the other MUC issues may have thrown it into an error state.

prosody_files {
    hosts = {
        -- each VirtualHost to be migrated must be represented
        ["domain.com"] = {
            "accounts";
            "account_details";
            "account_flags";
            "account_roles";
            "accounts_cleanup";
            "auth_tokens";
            "invite_token";
            "roster";
            "vcard";
            "vcard_muc";
            "private";
            "blocklist";
            "privacy";
            "archive";
            "archive_cleanup";
            "archive_prefs";
            "muc_log";
            "muc_log_cleanup";
            "persistent";
            "config";
            "state";
            "cloud_notify";
            "cron";
            "offline";
            "pubsub_nodes";
            "pubsub_data";
            "pep";
            "pep_data";
            "skeletons";
            "smacks_h";
            "tombstones";
            "upload_stats";
            "uploads";
        };
        ["conference.domain.com"] = {
            "accounts";
            "account_details";
            "account_flags";
            "account_roles";
            "accounts_cleanup";
            "auth_tokens";
            "invite_token";
            "roster";
            "vcard";
            "vcard_muc";
            "private";
            "blocklist";
            "privacy";
            "archive";
            "archive_cleanup";
            "archive_prefs";
            "muc_log";
            "muc_log_cleanup";
            "persistent";
            "config";
            "state";
            "cloud_notify";
            "cron";
            "offline";
            "pubsub_nodes";
            "pubsub_data";
            "pep";
            "pep_data";
            "skeletons";
            "smacks_h";
            "tombstones";
            "upload_stats";
            "uploads";
        };
    };

    type = "internal"; -- the default file based backend
    path = "/home/user/code/prosody-build/prosody-0.12.4/data/";
}

database {
    -- The migration target does not need 'hosts'
    type = "sql";
    driver = "SQLite3";
    database = "prosody.sqlite";
}

I do think eventually we will get there to have better support for non-ui or just backend apps so to speak.

We already have some apps that require a secondary domain on the front end, but that isn't necessarily the case for the backend. We just need Nginx to route things as expected by the clients, etc.

Glad you're here, packaging and interested in this stuff working.

I did some reading and it seems it is theoretically possible to make a server that utilizes different ports to differentiate the modules under a single sub-domain, but this would require tweaking and re-compiling an XMPP server.

Even if this was done, and all XMPP protocols followed, there's always the chance that some not-fully-compliant client that works with every well known XMPP server would not work with ours because we've chosen to deviate so far from the norm, while still technically following the standards. Making these upstream changes would require someone much more familiar with XMPP servers than me.

While I look forward to Cloudron allowing us to package more complex services, I think XMPP is in a pretty good place for the above-average admin; it currently takes less than 5 minutes to set it up. I will probably end up making a cron job to sync the certs into my app's storage volume when I deploy this to production. If I think it's useful to others, and Cloudron 9 isn't out yet, I'll share it here.

Something similar can be had?

@djxx, was the architecture chosen to use subdomains vs different ports?

If we're clever with the reverse proxy or with the main app server that listens to connections, it can ferry connections for other services by the request type or layer it needs to access, avoiding the need for extra ports or subdomains.

As for the HTTP page, why not make it multi-useful. For example, by default it's just a Cloudron OIDC login button and then per app it's configured what it does.. for backend services it could be a simple CLI stats page in HTML, or even a web terminal. That way it's flexible and useful to the admin, users, packaging and platform maintainers.

What else would be useful there?

Sorry that this is not yet implemented but we will get there in the future, so it is already great to collect requirements for such service type apps like prosody. This helps to avoid developing the wrong things on the platform side

Personally, I don't think it hurts that there's a 404 on an HTTP/S port for an application that doesn't serve HTTP. Putting up a big page like "You found an XMPP server!" just invites trouble from the bad people on the internet.

After reading up a bit on prosody docs, I still dont fully understand what we should do with the default location and what we could serve up as http (currently it shows a 404 error page for example). I saw there is also BOSH which is intended for http, so maybe that can be placed there instead, but so far I don't have a good grip on prosody's use of domains, certs and ports.

When is the next version scheduled for release?

All in all since we at least need changes to the TLS addon to also provide the root domain certs, we are looking at least to a package release after the next Cloudron version only. Just to keep expectations clear.

After looking again, it seems we could drop the "proxy" sub-domain since we're using Cloudron's TURN server (I would need to test). But there's still a non-zero number of extra domains needed, as well as the TLD cert.

I didn't realize that the TLD could be added as an alias which would give the cert. I was using /etc/certs before I did the hacky workaround to get the TLD. I will try the approach you mentioned and make sure things are still working.

One question is, apparently certs for the root domain are also required, even though no DNS records for that are?

Correct. The XMPP specification uses the SRV records to point to a specific server. It does need the cert to validate usernames like user@domain.com even if you chose to have the XMPP server somewhere else completely.

Otherwise there are naturally discrepancies elsewhere, given that this started (as far as I can tell) from some upstream Dockerimage which will not fit, but we can get through one by one.

Can you tell me a bit more about this? I changed it to use the Cloudron base, and confirmed that all the actions it does to build the server are successful. I did my best to follow putting data under /app/data, and runtime files under /run, according to the documentation.

For the extra required DNS records, we have to see if and how we can integrate this in the platform to support those. We already have some well-known type records, maybe it fits there.

This would be nice, but it's really not that difficult (compared to the entirety of setting up an XMPP server). I think if the package was installable and the only extra steps are some DNS changes, it'd still be very easy for people who want XMPP to do.

The alias domains still need to be manually setup by the user, so maybe we need some platform feature in the future for this.

Don't we already have this for e.g. apps like Loomio?

No need to merge that, just to give some idea about the certs handling. The alias domains still need to be manually setup by the user, so maybe we need some platform feature in the future for this.

For a start, you already added the tls addon, which does provide the certs in /etc/certs/ using that instead of the extra volume mount will ensure the app gets restarted and will pick up the correct certs if they get renewed. I am playing a bit from your package to hopefully get this working.

One question is, apparently certs for the root domain are also required, even though no DNS records for that are? This can be solved by also adding the root domain as an alias to the app.

For the extra required DNS records, we have to see if and how we can integrate this in the platform to support those. We already have some well-known type records, maybe it fits there.

Otherwise there are naturally discrepancies elsewhere, given that this started (as far as I can tell) from some upstream Dockerimage which will not fit, but we can get through one by one. The first I noticed was, that currently it doesn't work with the Cloudron app debug mode, since it uses ENTRYPOINT in Dockerfile, that was easy to change by relying only on CMD

• install the app to the 'xmpp' subdomain
• add extra DNS CNAME/A records pointing to the same server:
  • pubsub
  • proxy
  • upload
  • conference
• add these domains as aliases for the app
• add DNS SRV records:
  • _xmpp-client._tcp.domain.tld port 5222
  • _xmpps-client._tcp.domain.tld port 5223
  • _xmpp-server._tcp.domain.tld port 5269
• make a storage volume called "Extra Storage"
• mount the storage volume to the app
• copy certs into the storage under the subdirectory app_xmpp
  • for my server, the command looks like this (run in cloudron directly, not in the app terminal):
  • cp -f /home/yellowtent/platformdata/nginx/cert/* /mnt/HC_Volume_102134826/app_xmpp/
• restart the app to make sure it's starting up with all pieces in place (extra domains, DNS records, and certs)

Then you can check for compliance with these two tools:

• https://compliance.conversations.im
• https://connect.xmpp.net/

You would also need to add a few alias domains and DNS records manually.

cp: cannot stat '/media/Extra Storage/app_xmpp/*': No such file or directory

probably some oversight there, but don't have time to dig into this just now.