What's coming in Cloudron 6.3

[girish]

Pushed a change that makes the filenames of backups more readable. It's just app_<domain>_vVersion.

root@my:/var/backups/2021-05-09-000811-352# ls
app_test.cloudron.work_v1.1.3.tar.gz

[girish]

@girish said in What's coming in Cloudron 6.3:

Volumes - make mounting easier by automating fstab/exports entries

This is also mostly done. When adding a volume, you can choose the mount type

The current volumes are migrated as "no-op" mount type (as in, user managed the mount themselves). It shows the status of each volume as well:

One thing we decided to go with systemd mounts instead of /etc/fstab. This allows us to create mounts that have correct dependency with the unbound DNS server for CIFS and NFS mounts.

A similar mounting change will be done for the Backups view as well.

[jdaviescoates]

@girish said in What's coming in Cloudron 6.3:

One thing we decided to go with systemd mounts instead of /etc/fstab.

So what will happen to existing volumes that are already mounted using /etc/fstab?

[nebulon]

@jdaviescoates I think instead of coming up with migration code, which will be a bit messy given the fstab format to correctly parse in all circumstances, I think we will ignore those and ask the admin to reconfigure the volume via the UI once. That way the admin can test and validate timely.

[jdaviescoates]

@nebulon said in What's coming in Cloudron 6.3:

I think we will ignore those and ask the admin to reconfigure the volume via the UI once. That way the admin can test and validate timely.

Just to be clear, if the admin takes no action will existing volumes keep working?

[nebulon]

@jdaviescoates yes, the code does not touch existing mountpoints as such.

[girish]

We now show the ubuntu version is the settings view.

Additionally, there is now an alert for Ubuntu 16 users.

[micmc]

@girish AWESOME!

[A Former User]

This looks wonderful! Loving this QoL changes.

[d19dotca]

@girish Please tell me 6.3 is coming down the pipe this week. I'm so eager for it! Desperately needing some of these email improvements, specifically the most urgent for me is the ability to not forward email on to mailing lists if it's been identified as spam. The limitations currently are impacting the trust of my mail server by other systems like Gmail which is rate limiting my emails now because so much spam is forwarded on to a couple of Gmail addresses via the mailing list functionality. At least they're not outright blocking me, but that'd be the next logical step that I want to avoid!

[girish]

@d19dotca it's unlikely this week, we just had a call yesterday and decided to release what we have right now. So, we have already started testing and running e2e. I will leave a note on the progress here. I moved the security+email features to the next immediate release. As for the specific issue you are facing wrt not forwarding spam, let me see if the fix for that is easy and include it in this release itself.

[d19dotca]

@girish Oh sure, that'd be good. So there's going to be a bug fix version then I presume with what's already been done so far (such as 6.2.9 maybe or still 6.3.0)? And then email + security will be added to something like 6.4 instead if the next release is still 6.3?

[girish]

@d19dotca what's next is 6.3. the email + security will be 6.4 or maybe even part of cloudron 7. we are actually very close to multi-host with the changes in 6.3 !

BTW, about the spam fix, it's easy to check if an email is marked as spam and drop it (i.e based on spamassassin flags). However, since we don't have a place to quarantine yet, you will only see a eventlog entry that it's gone. Is this OK for the moment till we implement a monitorable mail queue?

[girish]

As a final update to the mounting logic, it is also integrated into backups and restore logic.

[scooke]

@girish said in What's coming in Cloudron 6.3:

Pushed a change that makes the filenames of backups more readable. It's just app_<domain>_vVersion.

root@my:/var/backups/2021-05-09-000811-352# ls
app_test.cloudron.work_v1.1.3.tar.gz

Thank you!!

[d19dotca]

@girish said in What's coming in Cloudron 6.3:

BTW, about the spam fix, it's easy to check if an email is marked as spam and drop it (i.e based on spamassassin flags). However, since we don't have a place to quarantine yet, you will only see a eventlog entry that it's gone. Is this OK for the moment till we implement a monitorable mail queue?

I think that's fine for now, but would it be possible to only affect mailing lists perhaps (assuming this change is mostly just temporary and probably will only really be used by me for the time being). I ask because I think if I were to just drop all messages marked as spam from all mail delivery completely, it'd be a detriment to my hosted mailboxes. The only place I want to disable sending spam mail is for mailing lists alone, if possible. But either way I guess, better than nothing I need to ensure Gmail isn't going to block mail from my server completely just because the two addresses my mailing lists forward to are total magnets for spam.

[subtlecourage]

@girish said in What's coming in Cloudron 6.3:

Make email setup inside apps optional. This will make it possible to configure specific apps to use some external service for mail delivery directly and the Cloudron package won't touch their mail settings.
Volumes - make mounting easier by automating fstab/exports entries

The email feature is something I think will help my use caase.
The volume feature is freaking phenomenal.

You are all amazing!

[girish]

Now that Cloudron is in the Vultr marketplace, it seemed the right time to implement Vultr DNS.

[girish]

The last change remaining in 6.3 was related to notifications and most of it had landed yesterday.

• Common issue is that a notification like a cert failure or restart required etc is essentially spammed to all admins and this causes great distress for all. So, we have now moved notifications from being per-user to be instead "system" level. The notification system is thus shared by the admin team and they can figure how to collaborate with other tools instead of Cloudron trying to have some sort of "read" indicator for each one of them.

• Apart from failed backups and cert renewal failures, no emails are sent anymore. You have to visit the dashboard to see notifications since most of them are really not urgent or actionable.

• Cert renewal notification + email is only raised if only 10 days are left to go for existing certificate to expire. The renewal itself will start 30 days in advance. So, we should see this very rarely from here on. Backup renewal notification will only be sent if 3 consecutive backups fail. This way we allow for "external services" to fail now and then without being too aggressive about notifying user.

There's more notes at https://git.cloudron.io/cloudron/box/-/commit/73917e95c9473ec8fc11216fd78607b0a256df7d . There's other minor visual changes as well, I will post a screenshot tomorrow once I complete the UI side.

[girish]

The going is slow... Having to support 3 ubuntu versions takes a lot of testing time The tests currently pass for ubuntu 18 and 20 but not for 16 yet. But we are close! Looks like failures are related to older versions of unbound not supporting sd_notify correctly.

[d19dotca]

@girish said in What's coming in Cloudron 6.3:

Having to support 3 ubuntu versions takes a lot of testing time

Maybe it's time to kill off support for Ubuntu 16?

[girish]

@d19dotca yes, from the next release, it won't be supported anymore. There is a notification to inform user to upgrade (I guess we should have had the foresight to do this the previous release itself..)

[girish]

The tests are passing . Now just a whole lot of manual testing next week.

[d19dotca]

Just thought I'd check-in on the status of 6.3.0.

[girish]

@d19dotca Manual testing of volume mounting revealed a lot of bugs, so we had to rework it a bit. The rework is done (as you can see from our git commits). I have started another e2e run just now.

Also, since we are now in the Vultr marketplace, we have gone ahead and added Vultr DNS and Vultr Object Storage as well in this release.

[girish]

Looks like we are all set. There's a few known issues which we will fix immediately - https://git.cloudron.io/cloudron/box/-/issues?milestone_title=6.3.1 . I will stage the release tomorrow morning and leave a note here.

[girish]

I pushed out 6.3.0. We hit a few migration issues, so it became 6.3.1. There are some minor issues being fixed, will be pushing 6.3.2 shortly. Please hold before you update to 6.3.

(to clarify, new installations get 6.3 but updates from 6.2->6.3 will give you a warning that it's unstable. please heed the warning ).

[msbt]

@girish what if some eager user already updated to 6.3.1, will those issues be resolved or will they run into problems?

[girish]

@msbt it should mostly be OK but there might be some regressions. Did you happen to hit any? Let me know so I can get it sorted out for 6.3.3 . So far, I haven't heard of any from the new installations and all the Cloudron we have access to seem to have updated OK.

[msbt]

@girish the only things that happened were the logo from the branding disappearing and the app-update icons were misplaced (top right corner ). After updating to 6.3.2 the arrows are back where they belong (although on the left side, but this is by design I reckon). The logo I just reuploaded. Other than that it all seemed fine

[girish]

@msbt those were the exact 2 regressions that were fixed there are some small issues in volumes which is getting fixed as well now in 6.3.3. the logo/branding is fixed in 6.3.2.

[d19dotca]

@girish - I see that the mailing lists aren't forwarding spam anymore and see it in the "Denied" categorized email logs which is great and helps a huge amount for many of my mailing lists. But... how do I overwrite this for certain mailing lists which are too sensitive to do spam filtering on? I don't see a way to modify this and one of the mailing lists are getting too many false-positives from one particular email I can't whitelist. Am I perhaps missing the checkbox to disable/enable the spam filtering for the mailing lists?

I have a bit of an urgent issue here that I need to resolve since I have a doctor who's not getting emails from a particular patient due to this. How can I resolve this?

Side note: It'd be awesome if the log entries showed which spam rules were triggered to mark it as spam, so that if it's a false-positive like it is then I can review and tweak as necessary. Also whitelisting emails and so many more things we really need in the email functionality. Please and thank you. I know the last few won't come until future versions, but in the meantime if I can somehow get around the mailing list spam filtering for a particular mailing list or if there's even a temporary way for me to whitelist/allowlist the email address being blocked, I'd really appreciate it.

[robi]

@d19dotca all for one? or one for all?

[girish]

@d19dotca There is not option to ignore spam rules per forwarding list. A work around is to simple create a mailbox and then create a forward (this forwards everything). Let me look into adding an option per mailing list for 6.3.4. This will take me a day or two, so hopefully the workaround of creating a mailbox and a sieve forwarding works in the intermiitent time.

[d19dotca]

@girish Yes please. Spam filtering should never be forced without any option to bypass it, so it should definitely have always been an option to toggle on or off for mailboxes and mailing lists (likely defaulting to on I suspect). Thanks Girish if you can add it in the next release.

[girish]

@d19dotca If the forwarding work around works for now, I will move this to the next release 6.4 along with all the email stuff. In fact, 6.4 will mostly be just email related changes.

[d19dotca]

@girish I'll have to see, in case that other issue is applicable still when I convert it back to a mailbox again where spam isn't really handled in any sieve filters. I'll try to get this sorted out in the coming hours and confirm. I'd suggest this may apply to others than just me though so the sooner the better to fix that please.

[girish]

@d19dotca spam still cannot be handled by user's sieve filters because the spam filtering is done before user sieve filters are run. so, for this specific address alone, you can just convert it into a mailbox and setup forwarding. the rest can remain as lists i guess.

(As for the fix, I am not sure about adding a "do not forward spam" option. I think it's best not to ever forward spam like it is now but make spam detection itself more configurable which I agree is quite rigid now. So, this is why it might take a bit more time).

[A Former User]

FYI it looks like, at least in Firefox, the update button might be inside the <a> tag for the app itself? It always opens the app in a new tab in addition to taking me to the update page.

[girish]

@atridad that got fixed in 6.3.2

[A Former User]

@girish Weird... I'll wait for a new app update an check again with cache cleared. It seemed to still be an issue with 6.3.3 but again, could be cache

[d19dotca]

@girish said in What's coming in Cloudron 6.3:

spam still cannot be handled by user's sieve filters because the spam filtering is done before user sieve filters are run

Just for clarity... are you saying Spam will never forward from a mailbox either then if marked as Spam by SpamAssassin since it "cannot be handled by users sieve filters"? Or are you saying if the mailbox is set to just redirect all messages that it'll still include spam message forwarding then too?

For what it's worth, until all the spam filtering improvements appear in a future release, it may be better to focus on making that optional for mailing lists until such spam improvements are made. Spam filtering should never be forced on anyone (I've made calls for that too when discussing Spamhaus for example), there should always be a way to bypass spam filtering when needed whether mailbox or mailing list, and that option should exist for any admins.

To be fair, I just re-read my original feature request and don't think I made it obvious at all it should have been an option, so that's on me as I may have unintentionally mislead you to thinking it should be a forced setting for all. It was definitely my intention for it to be optional. Similarly, I had also noted in that very same feature request "the ability to set a threshold number to avoid false-positives" specifically because if it's marked as spam to the mailing list it's then completely lost and can't be seen later unlike a mailbox that still keeps it in the spam folder for viewing, meaning it may be necessary to set a less-rigid spam threshold (from 5 default to say 8 for example) when it comes to rejecting mail to mailing lists.

Just some food for thought anyways.

In the meantime, I've created a mailbox again (moved away from mailing list) for this particular user and hoping that all messages (including spam) are redirected to them. Sorry again Girish for possibly misleading the feature request and having an issue right away, haha, I admit it's a bit of an edge case because of the sensitivity of this particular recipient's job (a doctor) which means that all spam filtering should be on their side, not mine which is why I needed that option.

[nebulon]

@atridad I fixed a different glitch there. However first I thought I got what you meant with the update indicator, but now I am a bit lost. Also neither in chrome nor firefox I can reproduce the link opening pattern you are describing.

[girish]

@d19dotca I think a mailing list not forward spam is correct (same way there is no option to not filter spam for mailboxes). I think what is missing is letting admins define what spam is and have more spam control (through this they can also define nothing is spam too). That said, I will see how much work all this is and decide on a short term checkbox or long term spam control.

[d19dotca]

@girish I agree that the spam filtering needs to be improved and in most cases it'd be fine to block spam to mailing lists (I basically wrote those feature requests after all haha), but that's part of the issue here too... in order for that new feature of not forwarding spam on to mailing lists to be effective and non-problematic (keyword there), we need those extra controls around identifying spam and setting all the different SpamAssassin options and such.

Until those extra controls arrive (it's understandable that'll take a while), the new feature can cause some unintended consequences for some since it didn't come with the ability to at least disallow spam filtering on a mailing list. There will certainly be cases where the spam filtering should be done on the recipients end, not Cloudron's end because at least if their spam filter catches it they can still see it in their junk box, but if the mailing list on Cloudron catches it then it's gone forever, so there's a bigger risk of Cloudron doing it for some types of accounts.

I would strongly encourage that ability to disallow spam filtering on mailing lists for those who need it at least until the better spam filtering is added to Cloudron along with all the other important mail features we've been clamouring for. Right now, that feature - while useful in probably 90-95% of cases - is also now a detriment to the remaining percentage without extra controls in place.

To be fair, maybe I'm misusing mailing lists though. I used them because it was much easier to manage for accounts which need to simply forward on to a different mailing address. Maybe I always should have been using mailboxes for that use-case? So I'll admit maybe part of the issue here is more the way I'm using it, but I still stand by the fact that the feature is a bit "half baked" without the ability to disallow spam filtering on them for sensitive accounts, and I hope that will be fixed until the time that the spam features are added down the road.

[girish]

@d19dotca I understand the impact of the change. Let me see if we have a few more hit this before making changes. It requires another full release cycle (a new mail container, new translations, new ui elements etc).

[d19dotca]

@girish Okay, thanks Girish for understanding. In the meantime I'm using the mailbox route again for the few addresses where I don't want to be doing spam filtering / preventing the recipient from receiving spam messages.

Relevant thread spin-off: https://forum.cloudron.io/topic/5277/spam-messages-forwarding-correctly-via-forwarding-sieve-rule/1

[nebulon]

@atridad said in What's coming in Cloudron 6.3:

FYI it looks like, at least in Firefox, the update button might be inside the <a> tag for the app itself? It always opens the app in a new tab in addition to taking me to the update page.

Actually I am seeing this issue now as well, not sure why it didn't happen when I tested this yesterday. Anyways I've pushed a fix for that now.

[luckow]

Something for the stats: 10 minutes for a Cloudron instance with 6 cores, 32 GB RAM, Ubuntu 18.04.2 LTS. 42 apps from Cloudron 6.2.8 to Cloudron 6.3.4

[subven]

@luckow exactly the same for me but with Ubuntu 20.04. No problems at all just a little waiting time. Watch the live logs to know what's going on during the upgrade.

[DanTheMan]

Problem for me after updating to v6.3.4 running on Ubuntu 18.04.4 LTS.
Matrix (Synapse server) seems to be running, but sending messages on Android and Windows not working anymore...

edit:
i moved this topic to https://forum.cloudron.io/topic/5291/problem-with-sending-and-receiving-message-s-android-windows

[nebulon]

@dantheman this maybe be better discussed in the matrix/synapse forum section as most likely this has nothing to do with the Cloudron update. If you have any further logs or error messages, please create a topic at https://forum.cloudron.io/category/50/matrix-synapse-element

[AmericanKulak]

This post is deleted!

[fbartels]

Hi,

a small ui glitch (at least it seems so to me) I just noticed in 6.3.4. When there is the update symbol for an app on the dashboard, clicking on it always brought you directly to the update screen in the past. In fact it still does, but additionally it now opens the app in a new tab.

[jdaviescoates]

@fbartels sounds similar/ the same as an issue @atridad had and I think @nebulon managed to track down

[girish]

@fbartels this is fixed but we have to make a new release.

[girish]

6.3 is now available to all. Please open a separate post in the appropriate forum section if you hit any issues.

[girish]

There will be a 6.3.6. It's going through the CI, it has many minor changes and also an important security fix (the last item)

[6.3.6]

• Fix broken reboot button
• app updated notification shown despite failure
• Update translation for sso login information
• Hide groups/tags/state filter in app listing for normal users
• filemanager: Ensure breadcrumbs and hash are correctly updated on folder navigation
• cloudron-setup: check if nginx/docker is already installed
• Use the addresses of all available interfaces for port 53 binding
• refresh config on appstore login
• password reset: check 2fa when enabled
• Fix crash due to wrong Spanish translation for invitation email

[mdreira]

@girish We did all the translation into Spanish. What was the problem and in what sentence?

[nebulon]

@mdreira there were just some syntax errors like extra whitespace for example:

<% = invite Link %>

a space between % and = as well as invite Link without the space
or

<%= inviteLink%>

and this also, where between inviteLink and % a whitespace was missing. Fixed version is now at https://translate.cloudron.io/translate/cloudron/dashboard/es/?offset=1&q=inviteLinkActionText&sort_by=-priority%2Cposition&checksum=

Most likely just autocorrect of your browser or copy and paste issue, so easy to fix once I found out. But would be nice if we could make weblate catch this somehow for us.

Ejs, which is the template engine we use here, is quite picky.