Cloudron Forum

Is it possible for some of us to create a Mattermost plugin which would allow the use of OAuth2 protocol? That way the additional functionality would be separated from core, (relatively) easy to maintain and could be extended to multiple services.

Hi Joseph,

My post above about MongoDB is not meant to close the thread. These are my three follow-up questions.

1. How do we reclaim the 2.46GB of reclaimable space?
2. Why on our 80GB Cloudron VPS do we only have about 30GB available for data? That's a lot of overhead.
3. Is there some way to organise this Cloudron VPS more efficiently? Is Cloudron stashing a day or week of backups somewhere on this VPS?

I look forward to your answers.

Cordial regards, Alec Kinnear

Hi Joseph,

You're right, it's not UptimeKuma. RocketChat is the web app which we run which uses MongoDB. We'll work on trimming it down.

Thanks, Alec

Hi Joseph,

Thanks for the tip about docker system df. Here's the results, which look more reasonable.

TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          17        17        21.5GB    2.436GB (11%)
Containers      22        18        0B        0B
Local Volumes   36        36        1.113GB   0B (0%)
Build Cache     0         0         0B        0B

1. How do we reclaim the 2.46GB of reclaimable space?
2. Why on our 80GB Cloudron VPS do we only have about 30GB available for data? That's a lot of overhead.
3. Is there some way to organise this Cloudron VPS more efficiently? Is Cloudron stashing a day or week of backups somewhere on this VPS?

Please answer all three questions if you could.

I'll get back to you about the MongoDB issue when I have more information.

Making the web work for you, Alec

Hi Nebulon and Girish,

I hope you gentlemen are doing well. How time flies when you're having fun. It's been more than five years of Cloudron for us.

We keep running into disk space issues with our Cloudron which is running on an 80GB DigitalOcean droplet. We've cleaned up NextCloud (one can't conveniently use the file sharing because of the backups which are difficult to fully delete) and Grav (which for some reason creates dozens and dozens of backups adding up to GB.

In total we have about 14GB of data in /home/yellowtent/appsdata, with another 8GB of mongodb in /home/yellowtent/platformdata. There's some other bits and pieces like logs but these are the big ones. Yet we have less than 15GB of space on our 80GB VPS.

What's left is the overlay2 directory (/var/lib/docker/overlay2). There's 75GB! of data stored there. I've seen that you've written that some of this storage is virtual (it must be as there wouldn't be room for the OS on our VPS otherwise, let alone any other data).

Still 75GB seems excessive and there's so little usable space.

I've run commands like docker ps -a, docker volume prune, docker system prune -a, docker image prune --all. This hasn't got me far. Here's all that was deleted:

root@my:/var/lib/docker/overlay2# docker system prune -a
WARNING! This will remove:
  - all stopped containers
  - all networks not used by at least one container
  - all images without at least one container associated to them
  - all build cache

Are you sure you want to continue? [y/N] y
Deleted Containers:
26796e79b1f631b60db7b03bad4f07b36d69769f73da559f080f1f25c4ef42e5
a5c57f2c3edffe486266949c6d82309aed7cc128895db305a972671c8d33ef6c
9f0dc548c88e9722014e94bb0137893fb16a690a76fc0a0f2df4c26f6e8c1055
b6b3ff484ec1306dc728a840375153b3dc2bad7fe0f2dc8e81136ce9339ae538

Total reclaimed space: 0B

1. What do we have to do to get more usable user space on our 80GB VPS?
2. How can we reduce the amount of storage used in /var/lib/docker/overlay2?
3. Data stored in our MongoDB seems to be very large. I suspect it's UptimeKuma history. Do you have any good tips on how to trim MongoDB in Cloudron down? A workflow, caveats/gotchas, safe way to reduce the MongoDB size?

Thanks in advance for your help. Please answer to the level in which I wrote. I.e. I'm an advanced user and not a programmer/full-time server admin. Please be explicit and not implicit with your instruction.

Making the web work for you, Alec

RocketChat is becoming really problematic. We only know RocketChat on Cloudron and we are running the 7.2 version. Our iOS apps don't notify reliably and don't keep a persistent login.

Desktop is much better.

We've noticed notifications about limits to the free version of 50 or 25 users (how is this FOSS?) but that doesn't affect us.

The steady degradation of RocketChat (we were delighted with RocketChat already at v3) has been frustrating. It's over ten years of RocketChat for us and it's now almost unusable. We hosted our RocketChat on the development team's paid servers, but the offer kept changing from flat $50/month to per user pricing (that would have affected us negatively at the time). What moved us off the RocketChat servers finally was the insistence on AWS infrastructure.

Keeping our private, open source chat application on the CIA's cloud providers didn't make much sense. It seems to me that outside of the commercial pressure, somehow the American intelligence community entered into a relationship with RocketChat where RocketChat gets some money/support in exchange for making sure everyone's RocketChat can be monitored. All RocketChat installs are federated with the development team server, which at least lets them know who is where and who is sending messages on a given server. That's a lot of information.

After ten years of RocketChat, five of which were very happy, not sure where to go now. Matrix, Zulip, Mattermost or even back to IRC. Or try to keep fighting RocketChat.

My point about fighting RocketChat is that Cloudron should probably be looking for a forked version of RocketChat which doesn't include all this monitoring software and limitations. I'll take a quick check now.

That's very good news, J. Thanks.

Deliverability to Microsoft and Google are a real challenge, that's for sure. Full DMARC with discard/delete options does help.

If we could just override the default Cloudron email configuration on a per app basis, it would be much easier to deal with

We're happy using our Cloudron Digital Ocean hosted droplet foliovision.net SMTP to deliver internal emails but can't use it for client emails (not reliable enough). Unfortunately Cloudron is missing a per application option not to hijack the app's email settings. This oversight caused us serious issues with InvoiceNinja where we had endless trouble with invoice deliverability due to the SMTP settings being silently erased on every restart.

@girish These issues with email being hijacked by Cloudron could all be solved by allowing a checkbox for each app not to touch the app's email configuration. I.e. email could be configured in the app. Our livelihood was threatened by Cloudron InvoiceNinja overriding our SMTP settings on every restart. This really should be fixed.

@nebulon said in Issue with Lengthy and Failing Updates:

This is of course no great answer for the moment. Not just specific to this, but 100s of Gb on top of various degrees of stable connections + remote storage protocols and endpoints which behave inconsistently in our experience (especially the lower cost they get) make this all not easy However there is much room to improve on our side with time.

Recommended solutions help. Backup works with this and this and this service. It's unreliable with that and that and that service.

Next big help would be for your backups to pace themselves for the long haul. You could pre-estimate size and refuse to run the backup in intervals which do not allow enough time to finish the backup (hourly, perhaps daily is not enough for some crazy Cloudroner with hundreds of gigabytes of data: we were just mailed by someone who uses our software with 260 TB of storage, argh, all at consumer level, not enterprise level).

Next you could cap backup speed not to max out the bandwidth for a working server.

Yes, there's lots Cloudron could do to make backups easier and more successful.

Ours fail all the time despite following guidelines. We had to manually remove 16GB of failed updates from our server (failed updates should autodelete), and then manually clean out NextCloud deleted files (which Cloudron could make automatic by default, even if NextCloud developers don't care about server space as they plan for dedicated servers and object space, our intrepid leaders at Cloudron could). And Cloudron didn't tell us we had to update from Ubuntu 18 LTS for backups and updates to work properly so we were caught in a closed loop. It's not easy setting up a service as complex as Cloudron but there is some low-hanging fruit here.

@nebulon said in RocketChat requires version 6.7 for app support but in back end we have Rocket.Chat 6.3.2:

Newer Cloudron versions should show when a platform update is blocked by an old Ubuntu version. We just can't update already shipped versions.

Glad to hear it. Not having that notification caused us endless trouble with Rocket.Chat over the last six months, including a couple of support requests to you. Your answers were effectively nonsense ("update rocket.chat"), as we could not update Rocket.Chat.

Thank you for the quick answer.

@girish said in RocketChat requires version 6.7 for app support but in back end we have Rocket.Chat 6.3.2:

Finally, if you are feeling brave :-), you can always do cloudron update --appstore-id appstoreid@package-version using the CLI tool to jump releases.

This is not a bad tip, I've only just gotten through all of the manual updates of Kimai, RocketChat, NextCloud and Uptime Kuma (Kimai is still happening).

If one of these multiple version upgrades go wrong, what's the option in terms of rolling back and moving more slowly on a single app basis?

Cloudron is still on deck for not providing useful notifications about when the underlying OS (Ubuntu) is blocking the updates.

@girish said in RocketChat requires version 6.7 for app support but in back end we have Rocket.Chat 6.3.2:

I wish VPS providers provided this as a service since they are best positioned to do this but they don't. In fact, Digital Ocean has completely broken Ubuntu 18 -> 20 distro upgrade already.

@girish What end users of Cloudron like Foliovision would like is to have a recommended hosting provider where Cloudron does everything to make sure everything works all the time (including underlying OS updates). Heck, if you can't do it with an external partner, I'm sure Cloudron could do well with its own servers if your pricing was in the range (could be slightly more expensive) than Digital Ocean/Linode. Most of us would trust Cloudron to do more to keep our data private than any of the US majors.

What's another huge nuisance is that I am required to update Rocket.Chat about 25x to get from 6.3.2 to 6.7.x. I understand that you like to force updates one .1 version at a time to avoid update issues, but for more popular apps, it would make sense to test and allow bigger update jumps, between major versions for instance. I'm also not sure if it's absolutely necessary to go from say 6.4.0 to 6.4.9 in ten steps.

@girish yes, you've hit the nail on the head. Since our Digital Ocean droplet Cloudron instance was on Ubuntu 18 LTS, very little was still updating.

We had quite a bit of trouble updating to Ubuntu 20 LTS (we didn't do the reinstall as recommended and Martin is still running around after MongoDB issues). What seems to be missing here would be:

1. notifications to tell the Cloudron owner/admins that updates are blocked by the version of the server
2. better integrated updates. We chose Cloudron as we don't want to administer these servers. Ideally Cloudron would update the core server and not require a full reinstall.

That said, we'll probably limp along under Ubuntu 20 LTS (not sure what's broken on Ubuntu 20, or rather blocked from updates, if you could send me to a list that would be great) until you and @nebulon have Cloudron 8 and Ubuntu 24 LTS under control (there will be teething pains, none of the improvements seem critical to us, we'd like the global email to be rolled back a bit, as with our complete fiasco on InvoiceNinja which still causes pain, constantly checking to make sure you haven't borked our email settings again and we have our hands full with our own open source software FV Player and other WordPress plugins to improve).

Anyway we'll move to Ubuntu 24/Cloudron 8 at some point and hopefully we'll enjoy four years of stability without blocked updates.

Hi @nebulon ,

I hope springtime is treating you and @girish well.

RocketChat requires version 6.7 for app support but in back end we have Rocket.Chat 6.3.2 in our Cloudron. This is with Automatic Updates enabled and even after a check for manual updates.

Is there something wrong with our Cloudron that it doesn't get the latest RocketChat versions?

Making the web work for you, Alec

Thanks @nebulon for the quick answer. It does look like you are shipping a more recent version than the internal version number suggests. We'll have another go at it tomorrow to try to figure out why the Cloudron version reports an earlier version number than
CustomContentBundle-2.2.0 and
CustomContentBundle-2.3.0 allow.

@girish I hope you are keeping well.

We use Kimai in production and would like to add the Custom Content plugin to improve some js quickly. We thought it would be an easy job and were delighted to pay for the plugin.

Imagine our disappointment when we managed to take down our Kimai server. It turns out that
CustomContentBundle-2.2.0 requires Kimai build 20100, while
CustomContentBundle-2.3.0 requires Kimai build 20500.

The Cloudron Kimai build is back at 20031 (a.k.a. 2.0.33). We have autoupdates enabled and manually updated to make sure we have the very latest Cloudron version installed. Strangely the package version is org.kimai.cloudronapp@2.1.3 which sounds like 20130 which would be enough to allow us to install
CustomContentBundle-2.2.0.

Could you look into the Kimai package and see if we can't get the latest version down the pipe? I know there were some breaking changes with v2 but these are not major version updates (that's behind us) but just improvements to Kimai 2.x.

Thanks, Alec

@nebulon said in Automated env configuration destroys InvoiceNinja custom mail configuration on every restart:

In an ideal world, we would probably hide that UI to avoid all this, but we are not the developers of those apps and contributing upstream

This is a backwards approach. It would be completely in Cloudron's control to check the .env variables for every app. If those variables change, Cloudron should:

1. notify the admin user with a screen on first startup alerting the user that key variables have changed (which ones)
2. send an email to the admin user alerting the user that key variables have been changed

I don't think local admins should have control over database settings but we should certainly be alerted if they are being changed back and forth.

It would also be very simple to allow users to enable or disable centralised email rewritten in each app. It's just one checkmark in the admin interface for the particular install.

That you have not spent the time to think about these issues and/or fix them is deeply disappointing. Your insouciance about how awful the situation is now is a deep indictment of the open source model (and this is from someone who has spent most of his adult life both creating open source and funding it).

Cloudron is sinking under too many apps and not enough care and housekeeping around the key ones you do support. I know the too-much-to-support issue first-hand. Due to WordPress making it ever more painful to keep our FOSS plugins both up to date and active in the directory (there were code change barriers, but even more administrative barriers), we had to retire about ten of them. Users can't find them or use them.

@girish does a great job running around to help us all and give us tips. It's outstanding. Systemic issues like this need to be addressed in a deeper way and not waved away with one blithe hand, or swept under the carpet.

Don't let the yes-men flatter you into complaisance. Unmerited pats-on-the-back is not how one improves a system or makes progress.

@necrevistonnezr said in Automated env configuration destroys InvoiceNinja custom mail configuration on every restart:

OneDrive or Dropbox would not require admin hours in a large organization - at a totally different price point

Dropbox requires almost no admin hours. If Condoleeza Rice had not been appointed to the Dropbox board in something like 2012 and if the Patriot Act had been repealed, Dropbox would definitely be the far better solution for filesharing between multiple computers. It's cheaper, easier and more reliable. NextCloud is fairly easily hacked (or backdoored through DigitalOcean) but it's not a systematic hack which gives the NSA and CIA direct access to all one's files, as if they are on Google Drive or Facebook or IMAP email.