Automated env configuration destroys InvoiceNinja custom mail configuration on every restart

[foliovision]

@nebulon said in Automated env configuration destroys InvoiceNinja custom mail configuration on every restart:

One consideration here is always how much options we support for how many apps. Each such tweak setting causes support tickets and ongoing testing, which takes time away from other things. Its easy to add initially but the long tail will have an impact.

Another consideration is the living hell we've been through with our InvoiceNinja instance. This is production financial software and Cloudron is willy-nilly overwriting settings.

Generally the Cloudron default email setup is unsatisfactory.

1. Cloudron recommends DigitalOcean as the default server provider.
2. Maintaining deliverability to large commercial ISP on DigitalOcean IP's is a nightmare/impossible (this is what we tried first, SendGrid only came in later).
3. Automated setup with SendGrid is not possible.

This is just tear-your-hair-out-in-frustration open-source-sucks territory.

Cloudron mysteriously overwriting our email settings has cost us about ten billable hours, disrupted client payments and communication and created a huge amount of stress for eighteen months.

Overwriting mail settings on every reset is about the craziest thing I've ever heard of. If you want to be this aggressive remove the email settings within the applications or lock them. Or warn us on every restart:

Your email settings have been overwritten. Please be sure you are happy with the current settings

and then show the current settings to us.

This libertarian attitude of it's up to you to know all your tools inside out all of the time, rather than a protect the user attitude, is one of the principal downfalls of open source (along with "make every option visible, fulfill every feature request however obscure").

There was a guy called Steve Jobs. He pioneered a method of software development which is to make applications and processes user-friendly. I call this way of designing software "intelligent defaults". Wiping email settings on restart is not an intelligent default.

Cloudron does so much of the plumbing right but this careless attitude towards email is extremely user-unfriendly, neckbeard kind of attitude, which does not fit the rest of what you get right.

Email must be just right for deliverability in 2023 (in fact any time after about 2018). Without perfect email pipelines, the production use of any communication application becomes a game of Russian roulette.

Right now there are certainly thousands of other Cloudron users going through frustration with email deliverability and email setup right now. Like us (and we are technically competent), they just don't know how they landed in the email hell they are in.

[nebulon]

Sorry if this particular issue caused you so much trouble. We are a very small team maintaining a large platform and a lot of apps with each their own special peculiarities. This is a huge field where we have to strike some balance between the vast use-cases amongst our user base. I doubt we can be compared to a company like Apple and that also not for the price point we have.

I am still not quite sure why you can't configure a Cloudron global email relay which will ensure deliverability across the whole system.

[foliovision]

@nebulon said in Automated env configuration destroys InvoiceNinja custom mail configuration on every restart:

I am still not quite sure why you can't configure a Cloudron global email relay which will ensure deliverability across the whole system.

1. Yes, we can. But it's a bit finicky with SendGrid. Moreover we only want InvoiceNinja on SendGrid at this point.
2. You still shouldn't be erasing people's email settings. This is just wrong. Particularly without actively notifying the user.

At the least, any app where you erase the email settings on restart should come up with a big notification on restart right across the screen that email settings have been zeroed with a hint of what to do about it. This would not be difficult to do across all applications as it's easy enough to run a check if email settings match env or not before overwriting them.

In terms of the platform and what you maintain, I agree it's a lot. It might be better if there were fewer applications with more robust support. Originally I spun up a Cloudron instance just to demo some open source software for internal use. We now use Cloudron in production. In production, issues like this are considerable friction and let down the user experience.

[necrevistonnezr]

Indeed, the documentation at https://docs.cloudron.io/apps/invoiceninja/ suggests you can edit the env file to your liking… At the very least, it should say what settings are not persistent

[girish]

@necrevistonnezr Customization within limits

I guess we expect users to know to not change authentication, database and email setup in apps directly. These 3 things are integrated into Cloudron since we identified these as 3 painful things to configure again and again when selfhosting apps. Database is very tied to backups, so this is close to impossible to change. Authentication is changeable only at install time. We tried to make this dynamic but couldn't since each app is special when it comes to migrating users from one auth provider to another.

Email can may be changable, still thinking about this after this thread. The reason why email is re-configured each time is because if you change domains of an app etc, the email is automatically re-configured on package start. A while ago we implemented the "Leave email configuration to the app" switch. We are open to add this switch to more and more apps as the use cases arise.

The original "expectation" that users need to know this has been a problem from day 1 We even put a splash screen immediately after Cloudron setup, but I think nobody reads those (can't blame them). Suggestions welcome...

[foliovision]

@nebulon Yes, simplification (I'm a fan) is always in conflict with customisation. But come on, the email preferences in InvoiceNinja are so clear and inviting. Billing is exactly the case where one might have special SMTP. Either lock these settings in InvoiceNinja (and show an error message when trying to save with information on where to change the information) or stop overwriting these preferences.

My vote in this case would be to stop overwriting the InvoiceNinja email preferences.

You have literally made my life much, much worse for two years with these decisions. There's been unnecessary friction with our most important clients due to missing invoices and deliverability issues. Cloudron became as much a source of frustration as joy due to this one issue.

Cloudron is not just a toybox at this point. People depend on it for work. Overlooking core issues like overwriting email preferences is no longer the right thing to do if it ever was. Cloudron is not proof-of-concept, it's production software.

[foliovision]

To follow up, some applications are just a pain-in-the-neck and are difficult to make work. For instance managing file space on NextCloud. NextCloud never wants to release files and delete them and it's very easy to run out of space and very hard to clear free space. These are complex issues and there are no simple solutions (apart from not running NextCloud which looks like it's probably our next step as the amount of time which goes into admin of NextCloud makes it a losing proposition). @girish was incredibly helpful and at least got us back to the starting line.

But in the case of InvoiceNinja, the application works, the preferences work. What was breaking InvoiceNinja was the Cloudron setup, not InvoiceNinja. Cloudron is effectively sabotaging effective work with an important app which does have its act together.

[robi]

@foliovision What was the procedure to properly clean NextCloud files? That might be useful to many here that use it.

[foliovision]

First one has to delete the files, then one has to make NextCloud delete the files (NextCloud only hides files), then one has to clean the database.

Here's our own notes:

We wanted to check what's taking so much space on the server and found the trashed items in NextCloud never seem to disappear. There does not seem to be any setting of NextCloud to do that. So you found a workaround:

First I removed content of /home/yellowtent/appsdata/230be9ee-72d6-4c54-8218-b08f8d217666/data/admin/files_trashbin on server
Then I run the command on NextCloud console on Cloudron to re-check files for the "admin" user:

sudo -u www-data php occ files:scan admin

[robi]

@foliovision is there a setting for nextcloud to not save trash data? Or only keep it for a short time?

[jdaviescoates]

@robi yes as per the first link in @foliovision previous post

[crazybrad]

I am fairly new to the Cloudron ecosystem, so my thoughts on email config may be impractical or not possible. In our experience (using Postmark), segmenting email streams is a desirable approach to improve deliverability. Different applications have different email rates, frequency, and loads and combining them into one outbound stream may be counterproductive.

Email platforms are intentionally vague about limits, best practices, what triggers spam classification, etc. But separating streams, at least in the Postmark ecosystem is their recommendation. My guess is other platforms would prefer that as well.

My suggestion (assuming this was possible) would be to support a global email config (default for all applications), unless an application-specific set of env variables existed. For those people who don't want/need outbound email streams by application, then the single, global email config would handle this. For those needing more, there would be one place in Cloudron to set all the email options, making it easier for us (and the support team) to debug.

[jagan]

The 'Bug' is an important feature. The other perspective is that of users like myself, who appreciate the magic of changing some of the values, say the 'sender email address', and the values in the env file are all automatically updated to reflect the new changed values, and everything just works.

But I understand your frustration if that is something you really don't want, and wish your customised SMTP values were left alone and not overwritten. To be fair, the common-denominator approach that suits most users is what is being followed. There will always be a need for even more customisation as the user base expands to greater degrees of heterogeneity.

[necrevistonnezr]

@foliovision This is not a general problem but must be specific to your setup. We have more than 100 GB on Nextcloud with various users and our admin folder is less than 16 MB (BTW for others: the uuid is unique, don‘t copy and paste the commands).

@foliovision said in Automated env configuration destroys InvoiceNinja custom mail configuration on every restart:

First one has to delete the files, then one has to make NextCloud delete the files (NextCloud only hides files), then one has to clean the database.

Here's our own notes:

We wanted to check what's taking so much space on the server and found the trashed items in NextCloud never seem to disappear. There does not seem to be any setting of NextCloud to do that. So you found a workaround:

First I removed content of /home/yellowtent/appsdata/230be9ee-72d6-4c54-8218-b08f8d217666/data/admin/files_trashbin on server
Then I run the command on NextCloud console on Cloudron to re-check files for the "admin" user:

sudo -u www-data php occ files:scan admin

[foliovision]

@necrevistonnezr said in Automated env configuration destroys InvoiceNinja custom mail configuration on every restart:

@foliovision This is not a general problem but must be specific to your setup. We have more than 100 GB on Nextcloud with various users and our admin folder is less than 16 MB (BTW for others: the uuid is unique, don‘t copy and paste the commands).

Okay, I'm annoyed now with myself for agreeing to answer an off-topic question in the conversation about email settings, which is critical. Our NextCloud setup is a test setup, somehow NextCloud managed to tie me into an admin user ID to be able to sync with WebDAV for iPhones to share address books. With individual user it wouldn't work and barely works with admin user. I've wasted days of my life on WebDAV sync for Apple Contacts but that's not the issue here, nor is our NextCloud. The point I was making about NextCloud is that it's a really complex application, whose vagaries are not the Cloudron team's fault. @girish and @nebulon do what they can to make NextCloud viable software (it's not, without serious server admin hours and a fair amount of study).

InvoiceNinja does work out of the box, and that it didn't work for us is mostly the Cloudron admin's team fault with this absolutely ludicrous zero the email setting on every restart.

Every day I think about how absolutely misguided this setup is, the more annoyed I become.

@nebulon, if you are going to zero your users' email settings on every restart, you certainly owe it to them to serve notice every time you do it. But mostly you shouldn't do it.

In fact, this situation is so easy to fix. If email settings have been changed in app, Cloudron has no business at all modifying those settings again. It's so brain-stunned an approach to providing open-source applications, I'm surprised no one has taken Cloudron to task for it before.

I'm not a server admin. I'm an expert on user interface and workflow, who is daily in contact with code and direct development. If Cloudron hopes to make any progress, you will have to improve your workflows and safety rails for people like me to be able to use your systems.

Another weak spot is the backups which always fail even when carefully configured. I had to get help within our office to solve the backups and the programmer in charge had to spend a fair amount of time to make them work with Linode object storage instead of fail and fail and fail again. At least we received notifications about the failed backups. We never got any notifications about how you were screwing with our InvoiceNinja settings and the how and why of it.

Go ahead @nebulon stick your head in the sand and blame user error again. This is a systemic failure, however clever it seems to you to zero people's email settings and/or force us to keep all apps on the same second-rate undeliverable SMTP, or upgrade all of our apps to external high end SMTP which requires a considerable management for subaccounts.

You're not running Microsoft or Apple where you can get away with ruining people's lives (deliberately breaking WebDAV every second OS release which worked fine and according to spec on 10.6.8) and continue to expand your business. This email settings fiasco should come to an active resolution and not just get swept under the carpet again, with "it worked for some people, it's how we've always done it".

It doesn't work and it's extremely user-hostile to erase people's custom settings. One more time, it's extremely user-hostile to erase people's custom settings. Don't do it.

[foliovision]

@necrevistonnezr said in Automated env configuration destroys InvoiceNinja custom mail configuration on every restart:

The uuid is unique, don‘t copy and paste the commands

That's not our uuid. 230be9ee-72d6-4c54-8218-b08f8d217666 is a random character string. And I really don't need to talk more about NextCloud on this thread. This thread is about InvoiceNinja and how Cloudron overwrites our custom settings with no notification.

[nebulon]

To take a step back here, the reasoning for resetting email configs for most apps, is that we take an approach where apps will ensure their Cloudron related settings on each restart to make sure the app is in a known state. The start.sh can be rerun and will always attempt to bring it to what we expect. This is to ensure a configuration in case a user has accidentally changed important configs, not just email but also database connection details, file permissions and the domain the app is running on. We mostly follow the same principle as ansible playbooks and the likes here. This is the only realistic way to deliver tested updates and be able to debug things efficiently.

Email configs though is not the same for all apps, as there are mainly two broad categories. One is transactional emails and the other is mass email like a newsletter. Those two require different strategies usually.

I guess the main issue with Invoice Ninja here arises, since we think InvoiceNinja falls into the transactional category, where deliverability is ensured via the Cloudron platform setting level. Which is why it gets overwritten on every restart. For newsletter apps, ideally they have two email configs for that matter, where the transactional will also get overwritten by our startup scripts and the mass mail sending goes via some relay suitable for this use-case.

So if you hit deliverability issues within InvoiceNinja, you likely also hit this in other apps. So having Cloudron wide email settings solved for this will also help with other apps.

Question now is why you have the need for having a custom setting for this one app, which in my view does not send out mass email?

From the UI perspective, I do agree, that this is not great that InvoiceNinja even allows you to configure email settings inside the app. In an ideal world, we would probably hide that UI to avoid all this, but we are not the developers of those apps and contributing upstream is currently often unfortunately not possible as we don't have the capacity and time for this. Hiring more people who can work on that aspect, would be great, but this is currently, also given the price-point, not viable.

[jdaviescoates]

My 2p:

• Cloudron is awesome.
• Their decision actually makes lots of sense for the vast majority of their users, in large part because...
• "make sure the app is in a known state" that @nebulon mentions is what makes backing up, cloning and migrating apps all remarkably easy and reliable.
• Backups actually work brilliantly most of the time. When they don't it's normally nothing to do with Cloudron, but instead to do with network issues or quirks of where one is backing up to.
• Nextcloud is evidently viable given it's by far the mostly widely used solution of its type and is used by millions of people from individuals up to large Gov'ts.
• If this has been a terrible pain point for 2 years, perhaps it should've been reported it 2 years ago.
• Being obnoxious is generally not a useful nor productive strategy for resolving issues.

That is all.

[foliovision]

@nebulon said in Automated env configuration destroys InvoiceNinja custom mail configuration on every restart:

Question now is why you have the need for having a custom setting for this one app, which in my view does not send out mass email?

I've answered that question above. Deliverability. I don't care about deliverability on the other apps, as they go to internal addresses and internal servers which are set up to accept Cloudron email (we have control). But that's not the essence of the question. The essence is Cloudron management sabotaging its users.

Stop tampering with our email settings @nebulon please. Your high-level view of how we should run our email is annoying and totally out of place. If the preferences are there don't overwrite them.

We do not want our email overwritten. On spinup, feel free to put some defaults in. On reinstall (if an app has been erased), feel free to put the defaults back in of course.

If I have changed settings within an app, do not touch them please.

I'd like to see change here, as the structure as it is now is simply wrong-headed.

I know you don't want to ruin your users lives @nebulon. But that is what you are doing by overwriting key settings without explicit warnings.

[foliovision]

@jdaviescoates You are welcome to your 2¢. I disagree with you and have enumerated exactly why. I have not said NextCloud is not viable. I've said running NextCloud is a huge waste of development/admin resources without a userbase of at least 20.