Cloudron Forum

Big update - I'm ready for your help @girish .

Repo: https://github.com/DerekJarvis/cloudron-prosody
Docker Image: derekjarvis/cloudron-prosody:20250406-042328-86345387d

What's working:

Everything.

This is a full-fledged XMPP server with:

• Healthcheck
• TURN server for jingle calling (voice and video)
• Uses LDAP for auth and automatically works for all users
• Supports sending/receiving files and images
• Supports message carbons (sync messages between devices)
• Supports standard (pretty) accounts like user@domain.com

(Direct TLS failing is intentional because it's deprecated and our upstream doesn't bother supporting it)

Approach:

I forked from a repository that has a good Prosody docker image with VERY secure defaults. I actually had to loosen a couple to make it more user friendly (like not requiring OMEMO). I added on the modules that make it even more useful (like HTTP upload for file attachments, TURN for jingle calls). I tried to make the majority of changes in our own files to make it easier to compare to upstream. I used Cloudron services (like LDAP, TURN) instead of running extra services in the container.

Help Needed:

There are a few VERY hacky things I had to do to make this work. Only one is a total blocker for real deployment by others, the others are all "nice to have" improvements.

TLD (Top-Level Domain) certs (blocker)

XMPP needs the TLD cert to allow the standard (and pretty) usernames like user@domain.com. Right now, there's no way for a Cloudron app to request the TLD cert. To work around this I did the following:

• Made a storage volume for the app
• Copied all the certs into the storage volume (yes, this is a very bad idea)
  • cp -f /home/yellowtent/platformdata/nginx/cert/* /mnt/HC_Volume_102134826/app_xmpp/
• Have the app use the certs it needs and then delete everything else

Aliases needed (nice to have)

The app is installed under the "xmpp" domain, and 4 aliases must be defined in the app, and also in the DNS records:

• conference.domain.com
• proxy.domain.com
• pubsub.domain.com
• upload.domain.com

It would be great if the Manifest allowed us to specify additional aliases the app needs, and also added the DNS records for those if the Cloudron instance is using a DNS API

SRV Records (nice to have)

XMPP uses SRV records to give information about the XMPP servers for a domain. Specifically:

• _xmpp-client._tcp.domain.tld port 5222
• _xmpps-client._tcp.domain.tld port 5223
• _xmpp-server._tcp.domain.tld port 5269

Add these manually, and your app will work fine.

It would be ideal if Cloudron allowed specifying additional DNS records that should be created if the Cloudron instance is using a DNS API.

Admin users (nice to have)

I couldn't find a way to identify Cloudron Admin users through LDAP. Prosody supports specifying an LDAP query to indicate who is an admin of the XMPP server.

It would be ideal if there was an LDAP query that could identify Admin users, and then Prosody would mark those users as admins as well.

External IP for checks (nice to have)

Prosody has a very convenient check function which will check for common errors. One of the things it checks for is if it thinks it is reachable by checking the DNS records it uses against the IP addresses it knows it has. So in the Prosody config you can tell it the known IP addresses for the server, and the check function will compare that to the DNS records to confirm the server is reachable.

It would be ideal if we have an environment variable for the server's primary IP address. With this the checks would not show so many false positives.

TLD environment variable (nice to have)

Prosody needs to know the TLD it is serving. I did not find an environment variable for this, so I hacked one using bash:
export DOMAIN=${CLOUDRON_WEBADMIN_ORIGIN#https://my.}

It would be ideal if Cloudron exposed an environment variable for the TLD.

Other Comments

The #1 reason I use XMPP is for self-hosted private chat. The #2 reason is it's how I have a virtual phone number for SMS. I use https://jmp.chat to get a VERY CHEAP sms number and use XMPP on all my devices to communicate with people through SMS on convenient chat apps like Gajim and Conversations.

I'm not sure if this is allowed, and if not, feel free to delete this post.

I wrote a blog post to share my journey of migrating my self-hosted NethServer to Cloudron. Overall I've been quite happy with the experience and wanted to share it here in case anyone else is considering the same thing, or if I might be able to answer questions for people experiencing some of the same pains during migrations.

If at least one poor soul considering the move from NethServer to Cloudron finds this post and it helps them, it'll make me happy.

I'm happy to say that I've moved my XMPP server from NethServer to Cloudron. While this is probably not a common move, I am sharing some notes here in case it helps someone else. Also, perhaps this'll cause Cloudron to show up in a few more searches.

• Install XMPP on Cloudron using the steps above. A bit manual for now!

• Dump your ejabberd data (that's the XMPP server NethServer uses) with this command:
  /opt/ejabberd-20.04/bin/ejabberdctl --config-dir /etc/ejabberd dump /etc/ejabberd/xmpp_dump.txt

• Download this dump file locally

• For ease, clone the source for prosody to your local computer so you can utilize the migration tools and not install needless packages on Cloudron. You'll need to run ./configure and ./make - but you don't need to actually install it.

• Don't be a Lua noob. I spent a while struggling to get my Lua environment setup, and thought I needed to run the tools like lua ejabberd2prosody.lua but got lots of errors about dependencies missing. Once I figured out you need to execute it directly like ./ejabberd2prosody.lua things worked fine.

• run the ejabberd2prosody.lua script on your xmpp_dump.txt file:
  ./tools/ejabberd2prosody.lua ~/Desktop/xmpp_migrate/xmpp_dump.txt

• Create a migrator configuration (or use the one I've pasted below). It basically takes everything from the file data format and puts it into the sqlite format, since that's how the Cloudron prosody is configured. Docs:

  • https://prosody.im/doc/migrator
  • https://prosody.im/doc/storage
     

• Run the migrator script:
  ./tools/migration/prosody-migrator.lua --config=./migrator.cfg.lua prosody_files database

• Turn off your Cloudron XMPP app

• Copy the resulting prosody.sqlite file into your Cloudron XMPP's /app/data folder. It will be in the /data folder under your local prosody directory.

• Turn on your Cloudron XMPP app

Your bookmarks, rosters, etc. will now be transferred to your new server! This doesn't appear to move archive messages (mod_mam). Probably because most prosody servers aren't configured to store these permanently so they don't bother migrating them.

I only noticed one issue while migrating. When I first ran the migrator script it gave me errors about topics being empty on some MUCs. I thought I was being smart and edited the code to handle the blanks. This caused me to be unable to join the MUCs on Prosody on certain XMPP clients because Prosody expects there to be a Topic for every MUC.

Once I manually adjusted the MUC topics to be non-empty, the other clients started working fine.

Another almost-issue is that Gajim needed to be restarted a few times to start using OMEMO properly. I think the other MUC issues may have thrown it into an error state.

prosody_files {
    hosts = {
        -- each VirtualHost to be migrated must be represented
        ["domain.com"] = {
            "accounts";
            "account_details";
            "account_flags";
            "account_roles";
            "accounts_cleanup";
            "auth_tokens";
            "invite_token";
            "roster";
            "vcard";
            "vcard_muc";
            "private";
            "blocklist";
            "privacy";
            "archive";
            "archive_cleanup";
            "archive_prefs";
            "muc_log";
            "muc_log_cleanup";
            "persistent";
            "config";
            "state";
            "cloud_notify";
            "cron";
            "offline";
            "pubsub_nodes";
            "pubsub_data";
            "pep";
            "pep_data";
            "skeletons";
            "smacks_h";
            "tombstones";
            "upload_stats";
            "uploads";
        };
        ["conference.domain.com"] = {
            "accounts";
            "account_details";
            "account_flags";
            "account_roles";
            "accounts_cleanup";
            "auth_tokens";
            "invite_token";
            "roster";
            "vcard";
            "vcard_muc";
            "private";
            "blocklist";
            "privacy";
            "archive";
            "archive_cleanup";
            "archive_prefs";
            "muc_log";
            "muc_log_cleanup";
            "persistent";
            "config";
            "state";
            "cloud_notify";
            "cron";
            "offline";
            "pubsub_nodes";
            "pubsub_data";
            "pep";
            "pep_data";
            "skeletons";
            "smacks_h";
            "tombstones";
            "upload_stats";
            "uploads";
        };
    };

    type = "internal"; -- the default file based backend
    path = "/home/user/code/prosody-build/prosody-0.12.4/data/";
}

database {
    -- The migration target does not need 'hosts'
    type = "sql";
    driver = "SQLite3";
    database = "prosody.sqlite";
}

@BrutalBirdie - thanks, but these are Node modules and not OS modules. Luckily, I did find a workaround. Step by step, I found all the missing modules, downloaded the deb files from https://packages.debian.org/, then extracted them into = /app/data/libatk/ (because that was the first missing library)

After this, I discovered the entrypoint script calls an editable script and edited /app/data/env.sh with these lines:

export PATH="/app/data/libatk:$PATH"
export LD_LIBRARY_PATH="/app/data/libatk"

Once I did this, I was able to enable Add Container Arguments on the Puppeteer nodes options in n8n and it worked just fine. The "container arguments" has it ignore all sandboxing best practices (which is OK since we're running in a container).

It would be good if the base n8n application for Cloudron would include the dependencies mentioned above, because my hack will get overwritten at each application update.

This ties into the following wishlist items:

• https://forum.cloudron.io/topic/7755/openfire-xmpp-server
• https://forum.cloudron.io/topic/2486/ejabberd-robust-scalable-and-extensible-realtime-server-using-xmpp-mqtt-and-sip
• https://forum.cloudron.io/topic/4188/snikket-server-your-own-messaging-server-in-a-box

These are the three "main" XMPP servers out there (with Snikket being a simplified deploment of Prosody). I think most of the people making these requests don't actually care which server is used - as long as it has the common modules, like roster, OMEMO encryption, file upload, etc.

I have chosen to package Prosody over the others because:

• OpenFire - I know nothing about it
• ejabberd - It's what is running on my server and the configuration for it is a pain
• Snikket - it's TOO simple, not giving us options to configure things

While Prosody does have its own docker image, I started from a different one because it has a sensible set of defaults baked in, is easy to extend, and is making use of environment variables for a lot of the configuration. Since Cloudron can inject environment variables with addons, I thought it would be easy to map Cloudron's environment variables to the ones expected by this container.

Here's where I'm at:

• I've made a repostory
• I've built a docker image
• I've tried to install it

Addons being used:

• tls - The app needs the certs to secure information on a number of ports other than HTTP
• ldap - I want to do SSO
• storage - The app needs to store data for message history, uploads, etc.

Upon install I'm getting an error related to me trying to create the cert structure described here: https://github.com/SaraSmiseth/prosody/tree/dev#ssl-certificates

mkdir: cannot create directory '/usr/local/etc/prosody/certs/upload.xmpp.domain.tld': Read-only file system

My plan for managing the SSL certs was to have the entrypoint script create the directory structure required by the app within the /app/data folder, copy the certs as provided by the tls addon, and symlink that directory to the expected directory /usr/local/var/lib/prosody. From what I've read, there are a few hard-coded things in prosody that may be difficult to change - which is why I'm trying to find a way to put the certs where it expects to find them.

So now I need some help figuring out the best way to deal with this read-only file system issue. Help would be greatly appreciated!

Other Notes:
There's a health check module for prosody that can also serve up a status through HTTP. I am planning to use this for the required healthcheck endpoint, and since an HTTP endpoint is generally not used for this application I also want to put it behind auth. The only purpose this HTTP endpoint serves is for the mandatory health check for the Manifest.

Small update - made good progress. What's working so far:

• XMPP server starts up, messages can be sent/received
  • this includes the pretty user@domain.com username
• LDAP Auth works, so users use their own accounts
• App health check

What doesn't work: anything related to HTTP endpoints (e.g. file uploads) . I've opened a ticket here: https://forum.cloudron.io/topic/13539/custom-app-httpport-not-being-proxied

Another datapoint. I tried using sshfs on my lan to the data server, and I got 112 MB/s write and 117 MB/s read - both of which are right at the theoretical limit of the gigabit connection. Not to mention it's taunting me with the read speed actually being faster than the write speed.

I decided to do another test with my laptop <-> hetzner server. So now we're transferring between the SSD on the VPS and the SSD on my laptop. The speeds are the same:
writing from laptop to VPS: 50 MB/s
reading from VPS to laptop: 7 MB/s

I checked and both my laptop and Cloudron are using the same version of SSHFS and Fuse:

SSHFS version 3.7.3
FUSE library version 3.14.0
using FUSE kernel interface version 7.31
fusermount3 version: 3.14.0

To get the best picture possible of the traffic, I put a fast watch on the connection:
watch -n 0 ss -t state established dst <server_ip>/24

With this, I can see that the write sizes are ~10x bigger than the read sizes. I'm wondering if this is why the performance difference is only seen in WAN situations; because 10x the round trips hurts a lot more on the WAN than the LAN. And for those of us with storage boxes in Europe and servers outside of europe, 10X the round trips really hurts.

I finally stumbled across this article (https://www.admin-magazine.com/HPC/Articles/Sharing-Data-with-SSHFS) which does some pretty detailed performance testing and tuning with SSHFS.

The options they suggest for the sshfs mount didn't help much. I did notice that using direct_io can sometimes make the read speeds go up to 20 MB/s, but it's not reliable.

I'm wondering if we're now into the realm of TCP configuration (which is the article's option #2) to increase a TCP buffer sizes. This would be a server-wide change, and is out of my depth. What are your thoughts, @nebulon ?

After a lot of banging my head, not to music - but against the wall, I've made progress. I was writing files into /app/data in the Dockerfile directly, which the docs has a small footnote saying this is a bad idea. Another "great" thing is that the location prosody searches for files is hard-coded at compile time, so I had to adjust that to get it to look in the right place.

The latest code is up now. The xmpp server starts, but it's not connecting properly. I think there are some cert issues that I'll need to spend more time debugging. At this point, I think it requires more Prosody expertise than Cloudron expertise.

@robi - I wonder if it still does any buffering when writing to /dev/null ? Since that's what the dd read command above does. In any case, this suggestion caused me to revisit the direct_io option. It says it disables the kernel paging cache, which does seem to give the most consistent performance improvement.

Yet Another Data Point - I did a lot more testing today, and I think I'm as far as I can go. The good news: I can consistently get 16 - 25 MB/s read speeds.

TL;DR: using this command gives me the best read performance (2x-3x improvement): nice -n -10 sshfs -s -o direct_io,compression=no

Why I'm using these options:

direct_io

direct_io disables caching, and had quite an interesting effect on reads.

Using the -f -d options I was able to watch the packets going through. I was wrong before about the writes being bigger than the reads; they're not. But the writes are being done more parallel than the reads.

Before direct_io:

[01315] READ
  [01308]           DATA    32781bytes (31ms)
  [01309]           DATA    32781bytes (31ms)
  [01310]           DATA    32781bytes (31ms)
  [01311]           DATA    32781bytes (31ms)
[01316] READ
[01317] READ
[01318] READ
[01319] READ
  [01312]           DATA    32781bytes (31ms)
  [01313]           DATA    32781bytes (31ms)
  [01314]           DATA    32781bytes (31ms)
  [01315]           DATA    32781bytes (31ms)

READ requests 4 chunks at a time, waits for them, and then requests 4 more.

[05895] WRITE
  [05827]         STATUS       28bytes (34ms)
  [05828]         STATUS       28bytes (34ms)
  [05829]         STATUS       28bytes (35ms)
  [05830]         STATUS       28bytes (35ms)
  [05831]         STATUS       28bytes (35ms)
  [05832]         STATUS       28bytes (34ms)
  [05833]         STATUS       28bytes (34ms)
  [05834]         STATUS       28bytes (34ms)
  [05835]         STATUS       28bytes (34ms)
[05896] WRITE
[05897] WRITE

WRITE requests at least 60 chunks at a time, and sometimes I saw over 100 chunks pending.

After turning on direct_io, the reads look more like the writes:

[06342] READ
[06343] READ
[06344] READ
  [06313]           DATA    32781bytes (31ms)
  [06314]           DATA    32781bytes (31ms)
  [06315]           DATA    32781bytes (31ms)
  [06316]           DATA    32781bytes (31ms)
  [06317]           DATA    32781bytes (32ms)
  [06318]           DATA    32781bytes (32ms)
  [06319]           DATA    32781bytes (32ms)
  [06320]           DATA    32781bytes (32ms)
  [06321]           DATA    32781bytes (33ms)
  [06322]           DATA    32781bytes (35ms)
  [06323]           DATA    32781bytes (35ms)
  [06324]           DATA    32781bytes (36ms)
  [06325]           DATA    32781bytes (36ms)
  [06326]           DATA    32781bytes (36ms)
  [06327]           DATA    32781bytes (37ms)

Note the difference in the chunk IDs and you can see it's allowing at most 31 chunks to be pending before requesting more.

I think this is the primary reason for the speed increase.

-s for single threading

I noticed that running it on a single thread caused the degredation of the repeated file reads to be less pronounced. Instead of dropping back to 8 MB/s after a few reads, it does 25 MB/s read at least 5-6 times (500-600 MB) before dropping down to 16 MB/s. Also, it recovers back to 25 MB/s over time, whereas with multi-threaded I needed to restart the SSHFS connection in order to get 25 MB/s speeds again.

nice

Since there seems to be an element of CPU bottleneck (as resolved by running in a single process) I also wanted to give this process priority. It seems to help the session get more 25 MB/s reads before slowing down

compression=no

Because we're now on one thread, and we're hogging lots of CPU time, I disabled compression. I didn't notice a difference in throughput with it on, but turning it off helps reduce CPU load

Next Steps:

I will run this test a few more times, and probably even adjust my mount for the volume manually to see if it helps performance.

There is definitely some element of throttling / filling up, because repeated reads in the same session can get slower, and starting a new session can help the speed go back up. I'm not sure if this is on the client side or the server side. Any insights would be greatly appreciated.

Even though I wish there was a clearer answer, I'll be happy if the 2x boost to read speed works.

P.S. - I even tried a "high performance SSH" binary hpnssh, and it did not make a noticeable difference in my tests.

Ok! I've taken this as far as I can go without some support from @staff . The app launches and mostly gets its certs right. Until the certs are fully correct I can't get it to actually run, start the healthcheck endpoint, or debug the rest.

XMPP typically expects to use the primary domain, e.g. mydomain.com . It uses SRV records to point to the specific sub-domain (if there is one) that should be used for XMPP. The reason for this is so XMPP ID's look the same as e-mails, like me@mydomain.com. This is also very convenient for XMPP that has LDAP enabled since people will type in their same username and password they do elsewhere. If you tell XMPP its domain is xmpp.mydomain.com, then user IDs look like: me@xmpp.mydomain.com, which isn't the same as e-mail addresses, and people often forget.

This is where the limitations come in. XMPP expects lots of sub-domains, like: upload, conference, proxy, and pubsub.mydomain.com. It also expects certs for each of these, and the primary domain.

The tls add-on only fetches a cert for the application's sub-domain. IF you are lucky enough to use a registrar that gives you API access, then you COULD use the wildcard certificate option which will cover all the sub-domains. However, this still doesn't give you the cert for mydomain.com which is required for the standard username me@mydomain.com .

There are a few adjustments needed to solve these problems.
1 - The application manifest should let an application define multiple domains. If the domain registrar API is being used it should set them all up at once.
2 - The application manifest should allow getting individual certs for multiple subdomains in the event a registrar API is unavailable for a wildcard cert. The user can configure the domains, but still let Cloudron get the individual certs. The tls addon should pass all of these certs through.
3 - The application manifest should allow other record types to be configured - like SRV records - so the registrar API can be used for a seamless install.
4 - The tls addon should allow an application to request the primary mydomain.com cert for cases like this.

@girish - As far as I know, all ports except 5280 don't serve HTTP - they're just TCP traffic. The 5280 port is just for some modules, one of which is a "status" module I planned to use for the required healthcheck endpoint. I did my best to put the data in the manifest file, but honestly I got the port information from here: https://github.com/SaraSmiseth/prosody#ports

As for the bare domain, yes - it just needs the cert. It won't actually communicate through this domain; It relies on SRV DNS records to point XMPP clients to the sub-domain that is actually serving XMPP: https://github.com/SaraSmiseth/prosody#dns

It's possible some of these ports and domains could be consolidated - but there will for sure be more than one domain, port, and cert needed to set XMPP up properly.

@girish Any update on this? I'd like to be able to keep tinkering on this before I forget all the prosody and cloudron stuff I crammed in my head

I will take another look at building the custom app, as well as the TLS option and report back when I'm ready.

I guess some clear set up instructions would help

• install the app to the 'xmpp' subdomain
• add extra DNS CNAME/A records pointing to the same server:
  • pubsub
  • proxy
  • upload
  • conference
• add these domains as aliases for the app
• add DNS SRV records:
  • _xmpp-client._tcp.domain.tld port 5222
  • _xmpps-client._tcp.domain.tld port 5223
  • _xmpp-server._tcp.domain.tld port 5269
• make a storage volume called "Extra Storage"
• mount the storage volume to the app
• copy certs into the storage under the subdirectory app_xmpp
  • for my server, the command looks like this (run in cloudron directly, not in the app terminal):
  • cp -f /home/yellowtent/platformdata/nginx/cert/* /mnt/HC_Volume_102134826/app_xmpp/
• restart the app to make sure it's starting up with all pieces in place (extra domains, DNS records, and certs)

Then you can check for compliance with these two tools:

• https://compliance.conversations.im
• https://connect.xmpp.net/

@robi I'm not an expert, but I did do some research on this to respond to Nebulon above. The main thing to note here is that the XMPP protocol expects different domains for different functions. So yes - it's an architectural choice that is mostly beyond our control. Each subdomain represents a different module, and the modules talk to each other (and other servers). They need a way to identify themselves, and they use the domain.

I did some reading and it seems it is theoretically possible to make a server that utilizes different ports to differentiate the modules under a single sub-domain, but this would require tweaking and re-compiling an XMPP server.

Even if this was done, and all XMPP protocols followed, there's always the chance that some not-fully-compliant client that works with every well known XMPP server would not work with ours because we've chosen to deviate so far from the norm, while still technically following the standards. Making these upstream changes would require someone much more familiar with XMPP servers than me.

While I look forward to Cloudron allowing us to package more complex services, I think XMPP is in a pretty good place for the above-average admin; it currently takes less than 5 minutes to set it up. I will probably end up making a cron job to sync the certs into my app's storage volume when I deploy this to production. If I think it's useful to others, and Cloudron 9 isn't out yet, I'll share it here.

Right as I typed that, I realized that there is the "send" icon next to it, which when clicking shows the link to set up the user's account. I realized that I could confirm the user's account on my own. I can initialize the account with the password I want, and then I also see the key icon you're referring to and am able to reset it that way.

Thank you for your help!

@girish - checking in again. I have another server I'm considering moving to Cloudron - but I need XMPP.

I'm facing the same issue with Nextcloud and trying to tune the performance of SSHFS (https://forum.cloudron.io/topic/13852/sshfs-read-speed-significantly-slower-than-scp-with-the-same-target/9) . The answer, at least for Nextcloud, is to adjust the configuration file. Instead of trying to "trick" Cloudron into accepting an SSHFS mount point as the primary storage, just adjust the applications config file to point to the mount point. Not sure if it will work for Immich, but it works for Nextcloud.

Also, I'm doing this with Nextcloud for the exact same reason - I want to manage my pictures. I'm trying the "Memories" plugin in Nextcloud which has prettty good reviews. I'll probably move on to Immich next for testing.

@girish - Given that the only blocker (that I know of so far) is the TLD certificate availability, would it be a decent workaround to package this entire thing as an application, and then run a one-time command to symlink the TLD certificate into the application directory? This would allow the sys admin to choose which application deserves access to the TLD certificate, ensure the app always has access to the latest certificate, and work around the current limitation of apps not being able to request the TLD cert.

@nebulon - Correct. You would need to make a volume called "Extra Storage", mount it onto the app, and have a folder under it called app_xmpp where you copy the sub-domain and TLD certs you need (I just copied all of them). This is an ugly workaround for the TLD cert not being available to the app through the manifest configuration.

You would also need to add a few alias domains and DNS records manually.