Cloudron Forum

That's very good news, J. Thanks.

Deliverability to Microsoft and Google are a real challenge, that's for sure. Full DMARC with discard/delete options does help.

If we could just override the default Cloudron email configuration on a per app basis, it would be much easier to deal with

We're happy using our Cloudron Digital Ocean hosted droplet foliovision.net SMTP to deliver internal emails but can't use it for client emails (not reliable enough). Unfortunately Cloudron is missing a per application option not to hijack the app's email settings. This oversight caused us serious issues with InvoiceNinja where we had endless trouble with invoice deliverability due to the SMTP settings being silently erased on every restart.

@girish These issues with email being hijacked by Cloudron could all be solved by allowing a checkbox for each app not to touch the app's email configuration. I.e. email could be configured in the app. Our livelihood was threatened by Cloudron InvoiceNinja overriding our SMTP settings on every restart. This really should be fixed.

@nebulon said in Issue with Lengthy and Failing Updates:

This is of course no great answer for the moment. Not just specific to this, but 100s of Gb on top of various degrees of stable connections + remote storage protocols and endpoints which behave inconsistently in our experience (especially the lower cost they get) make this all not easy However there is much room to improve on our side with time.

Recommended solutions help. Backup works with this and this and this service. It's unreliable with that and that and that service.

Next big help would be for your backups to pace themselves for the long haul. You could pre-estimate size and refuse to run the backup in intervals which do not allow enough time to finish the backup (hourly, perhaps daily is not enough for some crazy Cloudroner with hundreds of gigabytes of data: we were just mailed by someone who uses our software with 260 TB of storage, argh, all at consumer level, not enterprise level).

Next you could cap backup speed not to max out the bandwidth for a working server.

Yes, there's lots Cloudron could do to make backups easier and more successful.

Ours fail all the time despite following guidelines. We had to manually remove 16GB of failed updates from our server (failed updates should autodelete), and then manually clean out NextCloud deleted files (which Cloudron could make automatic by default, even if NextCloud developers don't care about server space as they plan for dedicated servers and object space, our intrepid leaders at Cloudron could). And Cloudron didn't tell us we had to update from Ubuntu 18 LTS for backups and updates to work properly so we were caught in a closed loop. It's not easy setting up a service as complex as Cloudron but there is some low-hanging fruit here.

@nebulon said in RocketChat requires version 6.7 for app support but in back end we have Rocket.Chat 6.3.2:

Newer Cloudron versions should show when a platform update is blocked by an old Ubuntu version. We just can't update already shipped versions.

Glad to hear it. Not having that notification caused us endless trouble with Rocket.Chat over the last six months, including a couple of support requests to you. Your answers were effectively nonsense ("update rocket.chat"), as we could not update Rocket.Chat.

Thank you for the quick answer.

@girish said in RocketChat requires version 6.7 for app support but in back end we have Rocket.Chat 6.3.2:

Finally, if you are feeling brave :-), you can always do cloudron update --appstore-id appstoreid@package-version using the CLI tool to jump releases.

This is not a bad tip, I've only just gotten through all of the manual updates of Kimai, RocketChat, NextCloud and Uptime Kuma (Kimai is still happening).

If one of these multiple version upgrades go wrong, what's the option in terms of rolling back and moving more slowly on a single app basis?

Cloudron is still on deck for not providing useful notifications about when the underlying OS (Ubuntu) is blocking the updates.

@girish said in RocketChat requires version 6.7 for app support but in back end we have Rocket.Chat 6.3.2:

I wish VPS providers provided this as a service since they are best positioned to do this but they don't. In fact, Digital Ocean has completely broken Ubuntu 18 -> 20 distro upgrade already.

@girish What end users of Cloudron like Foliovision would like is to have a recommended hosting provider where Cloudron does everything to make sure everything works all the time (including underlying OS updates). Heck, if you can't do it with an external partner, I'm sure Cloudron could do well with its own servers if your pricing was in the range (could be slightly more expensive) than Digital Ocean/Linode. Most of us would trust Cloudron to do more to keep our data private than any of the US majors.

What's another huge nuisance is that I am required to update Rocket.Chat about 25x to get from 6.3.2 to 6.7.x. I understand that you like to force updates one .1 version at a time to avoid update issues, but for more popular apps, it would make sense to test and allow bigger update jumps, between major versions for instance. I'm also not sure if it's absolutely necessary to go from say 6.4.0 to 6.4.9 in ten steps.

@girish yes, you've hit the nail on the head. Since our Digital Ocean droplet Cloudron instance was on Ubuntu 18 LTS, very little was still updating.

We had quite a bit of trouble updating to Ubuntu 20 LTS (we didn't do the reinstall as recommended and Martin is still running around after MongoDB issues). What seems to be missing here would be:

1. notifications to tell the Cloudron owner/admins that updates are blocked by the version of the server
2. better integrated updates. We chose Cloudron as we don't want to administer these servers. Ideally Cloudron would update the core server and not require a full reinstall.

That said, we'll probably limp along under Ubuntu 20 LTS (not sure what's broken on Ubuntu 20, or rather blocked from updates, if you could send me to a list that would be great) until you and @nebulon have Cloudron 8 and Ubuntu 24 LTS under control (there will be teething pains, none of the improvements seem critical to us, we'd like the global email to be rolled back a bit, as with our complete fiasco on InvoiceNinja which still causes pain, constantly checking to make sure you haven't borked our email settings again and we have our hands full with our own open source software FV Player and other WordPress plugins to improve).

Anyway we'll move to Ubuntu 24/Cloudron 8 at some point and hopefully we'll enjoy four years of stability without blocked updates.

Hi @nebulon ,

I hope springtime is treating you and @girish well.

RocketChat requires version 6.7 for app support but in back end we have Rocket.Chat 6.3.2 in our Cloudron. This is with Automatic Updates enabled and even after a check for manual updates.

Is there something wrong with our Cloudron that it doesn't get the latest RocketChat versions?

Making the web work for you, Alec

Thanks @nebulon for the quick answer. It does look like you are shipping a more recent version than the internal version number suggests. We'll have another go at it tomorrow to try to figure out why the Cloudron version reports an earlier version number than
CustomContentBundle-2.2.0 and
CustomContentBundle-2.3.0 allow.

@girish I hope you are keeping well.

We use Kimai in production and would like to add the Custom Content plugin to improve some js quickly. We thought it would be an easy job and were delighted to pay for the plugin.

Imagine our disappointment when we managed to take down our Kimai server. It turns out that
CustomContentBundle-2.2.0 requires Kimai build 20100, while
CustomContentBundle-2.3.0 requires Kimai build 20500.

The Cloudron Kimai build is back at 20031 (a.k.a. 2.0.33). We have autoupdates enabled and manually updated to make sure we have the very latest Cloudron version installed. Strangely the package version is org.kimai.cloudronapp@2.1.3 which sounds like 20130 which would be enough to allow us to install
CustomContentBundle-2.2.0.

Could you look into the Kimai package and see if we can't get the latest version down the pipe? I know there were some breaking changes with v2 but these are not major version updates (that's behind us) but just improvements to Kimai 2.x.

Thanks, Alec

@nebulon said in Automated env configuration destroys InvoiceNinja custom mail configuration on every restart:

In an ideal world, we would probably hide that UI to avoid all this, but we are not the developers of those apps and contributing upstream

This is a backwards approach. It would be completely in Cloudron's control to check the .env variables for every app. If those variables change, Cloudron should:

1. notify the admin user with a screen on first startup alerting the user that key variables have changed (which ones)
2. send an email to the admin user alerting the user that key variables have been changed

I don't think local admins should have control over database settings but we should certainly be alerted if they are being changed back and forth.

It would also be very simple to allow users to enable or disable centralised email rewritten in each app. It's just one checkmark in the admin interface for the particular install.

That you have not spent the time to think about these issues and/or fix them is deeply disappointing. Your insouciance about how awful the situation is now is a deep indictment of the open source model (and this is from someone who has spent most of his adult life both creating open source and funding it).

Cloudron is sinking under too many apps and not enough care and housekeeping around the key ones you do support. I know the too-much-to-support issue first-hand. Due to WordPress making it ever more painful to keep our FOSS plugins both up to date and active in the directory (there were code change barriers, but even more administrative barriers), we had to retire about ten of them. Users can't find them or use them.

@girish does a great job running around to help us all and give us tips. It's outstanding. Systemic issues like this need to be addressed in a deeper way and not waved away with one blithe hand, or swept under the carpet.

Don't let the yes-men flatter you into complaisance. Unmerited pats-on-the-back is not how one improves a system or makes progress.

@necrevistonnezr said in Automated env configuration destroys InvoiceNinja custom mail configuration on every restart:

OneDrive or Dropbox would not require admin hours in a large organization - at a totally different price point

Dropbox requires almost no admin hours. If Condoleeza Rice had not been appointed to the Dropbox board in something like 2012 and if the Patriot Act had been repealed, Dropbox would definitely be the far better solution for filesharing between multiple computers. It's cheaper, easier and more reliable. NextCloud is fairly easily hacked (or backdoored through DigitalOcean) but it's not a systematic hack which gives the NSA and CIA direct access to all one's files, as if they are on Google Drive or Facebook or IMAP email.

@robi said in Automated env configuration destroys InvoiceNinja custom mail configuration on every restart:

@foliovision is there a setting for nextcloud to not save trash data? Or only keep it for a short time?

Yes, but it looks like Cloudron defaults to keeping data forever, requiring those of us running NextCloud to dig very deep to figure out how to get rid of deleted files. Intelligent defaults here would be useful. NextCloud is such a bear that I don't blame Cloudron admins for throwing up their arms and telling users, "You want to run NextCloud? Learn how to admin it properly." InvoiceNinja is in another category. It's relatively easy to setup and easy to use.

Yes, users should not be changing database settings, that's too tied into the automation as @girish suggested. Totally understand. Email configuration can be set to default on first launch but then should not be tinkered with if the user changes email settings, particularly for apps where email is core to their functionality.

@nebulon already recognised this issue, which is why there are custom settings for email programs. nebulon should realise that email deliverability has changed in a cardinal way since he set up Cloudron. Email has gone from most email gets through unless there was a previous major spam issue from a given account to email does not get through unless the major ISP has been paid off (partnership relationships) or there is a full SPF/DKIM/DMARC setup, which is not entirely possible with Cloudron from what I can remember (we have setup what we can) AND you are not hosted on DigitalOcean (the recommended provider of Cloudron).

@jdaviescoates You are welcome to your 2¢. I disagree with you and have enumerated exactly why. I have not said NextCloud is not viable. I've said running NextCloud is a huge waste of development/admin resources without a userbase of at least 20.

@nebulon said in Automated env configuration destroys InvoiceNinja custom mail configuration on every restart:

Question now is why you have the need for having a custom setting for this one app, which in my view does not send out mass email?

I've answered that question above. Deliverability. I don't care about deliverability on the other apps, as they go to internal addresses and internal servers which are set up to accept Cloudron email (we have control). But that's not the essence of the question. The essence is Cloudron management sabotaging its users.

Stop tampering with our email settings @nebulon please. Your high-level view of how we should run our email is annoying and totally out of place. If the preferences are there don't overwrite them.

We do not want our email overwritten. On spinup, feel free to put some defaults in. On reinstall (if an app has been erased), feel free to put the defaults back in of course.

If I have changed settings within an app, do not touch them please.

I'd like to see change here, as the structure as it is now is simply wrong-headed.

I know you don't want to ruin your users lives @nebulon. But that is what you are doing by overwriting key settings without explicit warnings.

@necrevistonnezr said in Automated env configuration destroys InvoiceNinja custom mail configuration on every restart:

The uuid is unique, don‘t copy and paste the commands

That's not our uuid. 230be9ee-72d6-4c54-8218-b08f8d217666 is a random character string. And I really don't need to talk more about NextCloud on this thread. This thread is about InvoiceNinja and how Cloudron overwrites our custom settings with no notification.

@necrevistonnezr said in Automated env configuration destroys InvoiceNinja custom mail configuration on every restart:

@foliovision This is not a general problem but must be specific to your setup. We have more than 100 GB on Nextcloud with various users and our admin folder is less than 16 MB (BTW for others: the uuid is unique, don‘t copy and paste the commands).

Okay, I'm annoyed now with myself for agreeing to answer an off-topic question in the conversation about email settings, which is critical. Our NextCloud setup is a test setup, somehow NextCloud managed to tie me into an admin user ID to be able to sync with WebDAV for iPhones to share address books. With individual user it wouldn't work and barely works with admin user. I've wasted days of my life on WebDAV sync for Apple Contacts but that's not the issue here, nor is our NextCloud. The point I was making about NextCloud is that it's a really complex application, whose vagaries are not the Cloudron team's fault. @girish and @nebulon do what they can to make NextCloud viable software (it's not, without serious server admin hours and a fair amount of study).

InvoiceNinja does work out of the box, and that it didn't work for us is mostly the Cloudron admin's team fault with this absolutely ludicrous zero the email setting on every restart.

Every day I think about how absolutely misguided this setup is, the more annoyed I become.

@nebulon, if you are going to zero your users' email settings on every restart, you certainly owe it to them to serve notice every time you do it. But mostly you shouldn't do it.

In fact, this situation is so easy to fix. If email settings have been changed in app, Cloudron has no business at all modifying those settings again. It's so brain-stunned an approach to providing open-source applications, I'm surprised no one has taken Cloudron to task for it before.

I'm not a server admin. I'm an expert on user interface and workflow, who is daily in contact with code and direct development. If Cloudron hopes to make any progress, you will have to improve your workflows and safety rails for people like me to be able to use your systems.

Another weak spot is the backups which always fail even when carefully configured. I had to get help within our office to solve the backups and the programmer in charge had to spend a fair amount of time to make them work with Linode object storage instead of fail and fail and fail again. At least we received notifications about the failed backups. We never got any notifications about how you were screwing with our InvoiceNinja settings and the how and why of it.

Go ahead @nebulon stick your head in the sand and blame user error again. This is a systemic failure, however clever it seems to you to zero people's email settings and/or force us to keep all apps on the same second-rate undeliverable SMTP, or upgrade all of our apps to external high end SMTP which requires a considerable management for subaccounts.

You're not running Microsoft or Apple where you can get away with ruining people's lives (deliberately breaking WebDAV every second OS release which worked fine and according to spec on 10.6.8) and continue to expand your business. This email settings fiasco should come to an active resolution and not just get swept under the carpet again, with "it worked for some people, it's how we've always done it".

It doesn't work and it's extremely user-hostile to erase people's custom settings. One more time, it's extremely user-hostile to erase people's custom settings. Don't do it.

First one has to delete the files, then one has to make NextCloud delete the files (NextCloud only hides files), then one has to clean the database.

Here's our own notes:

We wanted to check what's taking so much space on the server and found the trashed items in NextCloud never seem to disappear. There does not seem to be any setting of NextCloud to do that. So you found a workaround:

First I removed content of /home/yellowtent/appsdata/230be9ee-72d6-4c54-8218-b08f8d217666/data/admin/files_trashbin on server
Then I run the command on NextCloud console on Cloudron to re-check files for the "admin" user:

sudo -u www-data php occ files:scan admin