What's coming in 6.0 (take 2)

[scooke]

@jdaviescoates @girish D'accord.

[jdaviescoates]

@girish said in What's coming in 6.0 (take 2):

You add a host path as a volume and give it a name. This is most likely some EBS/external hard disk/Block Storage.

I'm assuming volumes will support CIFS unlike current external data storage, right? (if not I had better cancel the Hetzner storage box I just ordered for this purpuse (or just start using it for backups where CIFS is already allowed)! )

[imc67]

@girish said in What's coming in 6.0 (take 2):

@imc67 Yes. Currently, the main restriction is that you can only add host paths under /mnt and /media as a volume. So, you can create some NFS mount or SSHFS mount there and then add it as a volume and then mount it into an app.

I really would appreciate it if you could create a step by step instruction in the docs by that time. I guess more users with large base “disk” would want to use this!

[mehdi]

@jdaviescoates said in What's coming in 6.0 (take 2):

I'm assuming volumes will support CIFS unlike current external data storage, right? (if not I had better cancel the Hetzner storage box I just ordered for this purpuse (or just start using it for backups where CIFS is already allowed)! )

The underlying problem is still there, CIFS / SMB don't support users and groups in the FS, so i guess it will depend on the apps, if they rely on FS permissions or not

[jdaviescoates]

@mehdi hmz, thanks, so I'm guessing likely not then.

So Hetzner storage boxes likely a no no for volumes (I guess I'll cancel the one I've just ordered, or just use it for backups)

I wonder what would be a good option...

I guess maybe Hetzner Volumes would work well, but they cost a lot more (a 500GB storage box is 5.68€/mo, whereas a 500GB Volume is 24€/mo)

[girish]

@jdaviescoates said in What's coming in 6.0 (take 2):

So Hetzner storage boxes likely a no no for volumes (I guess I'll cancel the one I've just ordered, or just use it for backups)

So, there's two kinds of storages:

• Data Directory - This is a location used by the app and only the app uses it exclusively. The app will store say attachments, profile pics, avatars, plugin code and whatever it needs in this directory. Data directory has to be ext4 because the packaging code requires file system permissions to work etc. This is the location you specify in https://docs.cloudron.io/storage/#app-data-directory . You can think of this app as part of an app's "state" (just like the database) and it is thus backed up and restored.

• External Directory/Volumes - This location can be any file system. Mostly the app just reads file (docs, music, movies etc) from this. So the permissions don't matter. It is also expected that these directories can be shared across apps. This is similar to the home directory on linux or My Documents on Windows. It's just separate directory that apps have access to. If you uninstall, backup apps, that folder is untouched. For these kind of directories, the file system type is not relevant. It can be anything - sshfs, cifs, ext2 whatever.

[jdaviescoates]

@girish OK, thanks, that's clearer.

However, some of the music streaming apps etc like Navidrome (which is what I thought I'd use a Hetzner Storage box CIFS for) can have users with different permissions too, so still not sure if that'll be possible?

Although the way I'd actually want to use it is that only one person (me) will be able to upload music, and all other users will just have read access, so I guess that could work fine?

[girish]

@jdaviescoates said in What's coming in 6.0 (take 2):

However, some of the music streaming apps etc like Navidrome (which is what I thought I'd use a Hetzner Storage box CIFS for) can have users with different permissions too, so still not sure if that'll be possible?

Yes, definitely possible. App users are entirely different from file system permissions. Without getting too technical, users inside an app are just "virtual users". They just exist inside the app's database.

[jdaviescoates]

@girish OK!

So maybe I will start uploading my 290GB of music up to my new 500GB Hetzner Storage Box to use like this as soon as 6.0 goes live!

[jdaviescoates]

@girish said in What's coming in 6.0 (take 2):

see the WIP tag

Oooh, just looked at that for the first time in a while and I see BigBlueButton and Jitsi Meet are included... who is actively working on them atm, is it you @girish @nebulon ?

Cloudron 6.0 with full text email search, volumes and BBB/ Jitsi would be SO awesome! (loving all the new music/ media apps too!)

[mehdi]

Once there are proper Volumes, my River app won't have too much reason to exist, so you can expect a few other apps soon after 6.0 release as I split river into independent apps (namely SickChill, CouchPotato, and Transmission)

[Lonkle]

@mehdi said in What's coming in 6.0 (take 2):

Once there are proper Volumes, my River app won't have too much reason to exist, so you can expect a few other apps soon after 6.0 release as I split river into independent apps (namely SickChill, CouchPotato, and Transmission)

Did your River app combine those apps with an added external storage connection bit?

[mehdi]

@lonk It combines Jellyfin (for which there's already an app now), Transmission, SickChill, Couchpotato (the 3 I just mentioned), a custom file manager (built before Cloudron had one), and a custom TV Shows and Movies streaming interface (built before I integrated Jellyfin). Plus a few custom things, like a script to auto-remove finished torrents on transmission, and a script to auto-convert videos to MP4 for easy streaming in the browser (for my custom streaming interface. I disabled it now that I mainly use jellyfin, which handles this automatically).

[girish]

@mehdi said in What's coming in 6.0 (take 2):

Once there are proper Volumes, my River app won't have too much reason to exist, so you can expect a few other apps soon after 6.0 release as I split river into independent apps (namely SickChill, CouchPotato, and Transmission)

When you have time, please split them out and I can start approving them as unstable already. It also gives me a good test bed to test the volumes stuff across apps and permissions.

[mehdi]

@girish said in What's coming in 6.0 (take 2):

When you have time, please split them out and I can start approving them as unstable already.

OK, will do. I will have to pick your brain a little about how to handle authentication for these apps. In river, they're behind a custom auth proxy, as they don't handle auth by themselves.

[girish]

@mehdi said in What's coming in 6.0 (take 2):

OK, will do. I will have to pick your brain a little about how to handle authentication for these apps.

Would it make sense to bring that into box code somehow? Maybe we add some flag in the manifest to turn on this "authentication wall".

@nebulon Is it possible to have a login screen like the surfer app but served from box code? I guess session management possibly won't work since app has no clue about this? Maybe logout won't work as well since there is no logout button in the app.

[mehdi]

@girish said in What's coming in 6.0 (take 2):

Would it make sense to bring that into box code somehow? Maybe we add some flag in the manifest to turn on this "authentication wall".

It would be possible yeah. An other possibility would be to have a standard way to do it in the "base image", to make it easy to implement in the apps that need it. I would probably go with the "base image" path, because it would be easier to configure on an app level IMO.

[girish]

@mehdi Good idea. Maybe then for the moment we can copy/paste apache configs in every app (like https://git.cloudron.io/cloudron/simple-torrent-app/-/blob/master/apache/cloud-torrent.conf ) for now and once we have a something common, I can try to put it in the next base image.

[Lonkle]

I really like that approach as well. I'll follow what you guys create and what changes occur to the base image! ️

[nebulon]

So the idea is to put some kind of "framework" into the base image, which can be used by apps? Wouldn't that anyways still not mean that an app needs to be patched for at least the logout action? Also would we do this as a php set of features? I do like to not pull this into the platform code as such, as that does not increase dependency on that.

Alternately, we could certainly add a login screen served up with some kind of session. The question then, as already mentioned, is how to logout. We could provide the app with a logout link, still that needs patching the app to some extent.

[mehdi]

@nebulon IMO the login part is much more important than the logout part. We can even do completely without the logout at all in the interface, with just a /logout URL that one would have to enter manually (if ever).

As to the precise tech to use, I already have a working version in Node.JS in river that I could isolate. If you guys prefer to re-do it in PHP instead or something else, it's your choice.

[nebulon]

Right, I was only bringing up php since that might be more commonly already be used within such apps, I would prefer a nodejs solution though. Maybe we can collect some arguments for and against adding to the base image or into the platform.

If we add it to the platform, we could have it more easily streamlined with the Cloudron look and feel, however within the base image the app could style it more towards its own look and feel. Putting it in the platform on the other hand would allow support translation now once it fully landed. Also say we use a nodejs based version, then we have to keep running an additional process with possibly another proxy even?

[jimcavoli]

Just pointing out that this might be better served by using something like a proper gateway and load balancing solution in front of the apps like Kong rather than NGINx on the box. An add-on for a basic auth screen could just be a config tweak to the box ingress which hits an auth wall - added advantage would be much more flexible routing to apps (sub-path, etc.) and flexible options like having multiple DNS names resolving to the straight A record for things like apps that can serve multiple domain names off one instance.

[robi]

Bringing in some service mesh concepts would be interesting.

[MooCloud_Matt]

@jimcavoli
You mean to use Kong or Traefik (this are exeple) as incoming proxy for every container ?

If yes, we need to consider how reliable is Nginx and the feature that can be easily added (Proxy_Cache, brotli, WAF, ...)

[jimcavoli]

@moocloud_matt yeah, there's currently a box level NGINx proxy - my idea is to replace that with a very hand-wavy something else which may be capable of shimming authentication for those things that just don't have it inbuilt (Kong) or if we go a different route on that sort of thing, we could use Traefik or similar at that layer. I think the use cases are intriguing enough to at least try the Kong route and fall back to something like Traefik if need be

[MooCloud_Matt]

@jimcavoli
for adding .htpasswd support ? or what kind of auth are you talking about?

[jimcavoli]

@moocloud_matt Kong is a pretty expansive topic on its own, but the idea would be with it in charge of ingress/routing, you could have a simple add-on config that would take care of enabling a plugin like https://docs.konghq.com/hub/kong-inc/ldap-auth/ on the route to a particular app, so you could have HTTP basic auth, but completely backed by the full Cloudron user store for any app that doesn't have its own scheme, providing clean headers that could be easily handled by that app's eb server or whatever

[girish]

Kong is indeed a separate complex topic. I think for the moment, if we had some template that people can quickly copy over to the app to get auth screen/login, it will help already. @nebulon do we have such a template already ? (like the one we use for our internal apps).

[MooCloud_Matt]

@jimcavoli
i think that's possible with nginx too, the ldap backend for auth.
Custom Template for nginx config, will be the best i think, especially for performance optimization.

But this Kong proxy is interesting i will ask to my team, if they have use it.

[nebulon]

@girish said in What's coming in 6.0 (take 2):

Kong is indeed a separate complex topic. I think for the moment, if we had some template that people can quickly copy over to the app to get auth screen/login, it will help already. @nebulon do we have such a template already ? (like the one we use for our internal apps).

Yes in various shapes, but all nodejs based. This would be trivial to add, however if many of those apps are just apache+php does it make sense to add supervisor+nodejs+someproxy to those apps just for a login screen?

[avatar1024]

@girish quick question in the mailbox sharing feature. If you make it so that a single inbox can have multiple owners (great feature btw), do you think it would then be possible to have the option to set a group as the owner so the mailbox ownership gets dynamically updated with changes in group membership? Thanks

[girish]

@avatar1024 Yes, that's the idea. The ownership will be dynamic.

That said, the initial outlook for the feature is not looking so good. There are two issues that need to be sorted out (suggestions/ideas welcome):

• Apps like SOGo show the "display name" of the user in the main UI. With a shared mailbox, it's not clear where this name should come from. With a single user, we give SOGo, the user's name. With multiple users, it's not clear what this should be.

• The authentication (from a user's point of view) is a bit confusing. Or maybe it's not, I would welcome some feedback here. You have to authenticate with the user's username/password but use the shared mailbox as the mailbox name. In some ways, this is the case already, when you use a different mailbox name with a different username.

[avatar1024]

@girish thanks for the reply.

Perhaps there are a few things I do not understand about the technical implementation of such a feature but I will still try to give my opinion. As you describe in the first post on this thread the idea is, rather that a "shared mailbox" as such, to have a single mailbox with multiple owners, so:

• for point 1) I suppose you mean that the name of the "owner" is fed to the app when the owner is assigned to the mailbox rather than when the user logs in? If so, would it possible to feed the name of the user that logs in into that (which will be recognised through its unique pair of credential: mailbox name + its unique password). If not then I'd say, as a first implementation of this feature (which could be improved later), then the name of the mailbox (as in the prefix before the @) should be fed to the app as the Display Name, when more than one owner are set to that mailbox

• for point 2, then yes, as you say the username (for login) should just remain the full mailbox email address (just that in the case of a shared one, one username will but associated to multiple passwords as valid login credentials).

[avatar1024]

@girish Or as I suggested in my previous post, if we could assign a "group" as the owner of a mailbox, then group name could be fed as the Display Name (for point 1) and you potentially use the group name for the login username (for point 2).

@avatar1024 said in What's coming in 6.0 (take 2):

...be possible to have the option to set a group as the owner...

[girish]

@avatar1024 said in What's coming in 6.0 (take 2):

@girish Or as I suggested in my previous post, if we could assign a "group" as the owner of a mailbox, then group name could be fed as the Display Name (for point 1) and you potentially use the group name for the login username (for point 2).

Making shared mailbox feature work only with groups is an excellent idea! I have to try this out and get back on how well this works.

[Lonkle]

@moocloud_matt said in What's coming in 6.0 (take 2):

@jimcavoli
i think that's possible with nginx too, the ldap backend for auth.
Custom Template for nginx config, will be the best i think, especially for performance optimization.

But this Kong proxy is interesting i will ask to my team, if they have use it.

Should we branch this topic off into its own thread about centralized authentication? It seems like an important aspect to discuss but this thread is about 6.0, which this couldn't be a part of, right? This would be further down the road? @girish @nebulon

[avatar1024]

@girish Glad it was at least a useful suggestion. Let's see if it works in practice

[girish]

@avatar1024 That worked out quite nicely. You can now select a group as owner of a mailbox. founders and sales-team are groups in the screenshot below.

[marcusquinn]

@girish Will each person's password work for IMAP/SMTP? (users in the group)

[girish]

@marcusquinn Yes, that's the idea. Each user can use their own password for sending/receiving with same mailbox.

[A Former User]

@girish Just a suggestion, but IMO it should have two dropdowns like the access section for apps. One for users and one for groups. Unsure of the implementation details but that just stood out to me.

[girish]

@atrilahiji Currently, one can only choose a user or a group. This was just the initial UI anyway for testing the backend. I think we probably need select box with groups like

[marcusquinn]

@girish makes sense. Be good as an input-select box, so you can partial-type what you're looking for in the case of long lists.

[girish]

@atrilahiji That was easy.

@marcusquinn Yes, it's searchable like the one in user's/group's view. The search only appears with more than 5 users.

[ei8fdb]

@girish said in What's coming in 6.0 (take 2):

There is a section overload in the app configure view We are looking to reducing the list of sections in the left pane. It's making things hard to find.

Hi @girish I'd like to help with this if you're interested.

[ei8fdb]

@girish said in What's coming in 6.0 (take 2):

We haven't decided on the naming yet, but maybe Volumes can be instead renamed to External Media or something, if it clarifies things. Open to other suggestions as well.

External media makes sense to me. Until the point Cloudron is able to backup these up (if that's something that will happen), I'd also have a tooltip saying 'External media is not included in Cloudron backups'.

[ei8fdb]

This post is deleted!

[girish]

@ei8fdb said in What's coming in 6.0 (take 2):

Hi @girish I'd like to help with this if you're interested.

Thanks for opening that gitlab issue! We won't do it for the current release atleast since there is massive UI surgery going on already for translations. Let's pick this one up after Cloudron 6 release in a separate thread.

[girish]

An update on this. The i18n is going along well, I think @nebulon is ~80% done. Once we are almost there, we will let you guys know in the translations thread. There is a large number of strings to be translated. We plan to ship Cloudron 6 with the code being translatable while the actual translations will be made available whenever they get ready.

Email container now has solr integration! While search is fast and I can search body (and maybe in future release attachments as well), there is something strange about how IMAP searches work. It seems it will search only under INBOX without the mail client having support for ESEARCH IMAP extension. I am also trying to figure how various mail clients behave here. Hopefully, should get sorted out this week.

Other features are done.

[jdaviescoates]

@girish said in What's coming in 6.0 (take 2):

An update on this. The i18n is going along well, I think @nebulon is ~80% done.

Excellent and important work!

Once we are almost there, we will let you guys know in the translations thread. There is a large number of strings to be translated. We plan to ship Cloudron 6 with the code being translatable while the actual translations will be made available whenever they get ready.

Good plan

Email container now has solr integration!

Amazing!

While search is fast and I can search body (and maybe in future release attachments as well),

Attachments too would be excellent!

there is something strange about how IMAP searches work. It seems it will search only under INBOX without the mail client having support for ESEARCH IMAP extension. I am also trying to figure how various mail clients behave here. Hopefully, should get sorted out this week.

I could be way off mark here, but I've noticed in the past that lots of IMAP providers seem to put Sent and other folders as effectively subfolders of the inbox. Perhaps something to do with that?

Other features are done.

You're underselling yourself (tbf, that's always a good look).

Personally I am very exciting about these "other features"!

• Volumes!
• Mailbox sharing!
• Faster Nextcloud and WordPress!
• Ubuntu 20.04 support! (Edit: tbh not particularly excited by this, because I'm more excited about having BBB on Cloudron and that isnt' even compatible with 18.04 yet!)

These are all fantastic new features!

[MooCloud_Matt]

@jdaviescoates said in What's coming in 6.0 (take 2):

Faster Nextcloud and WordPress!

We have some test image with Nginx + WP, that i have already shared with @girish, in the next few day we will work with him on some special feature to make the manifest compatible with OLS base image that we have pretty much ready, and later we will work on WP and NextCloud performance boost.

Probably next week, we will post some performance benchmarks of the new WP image, for both Nginx and OLS base image.

[robi]

@moocloud_matt oh you beat me to it.. was going to do Nginx based stack then add other webservers to the mix.

Can't wait for your testing.. godspeed.

[avatar1024]

@jdaviescoates said in What's coming in 6.0 (take 2):

Edit: tbh not particularly excited by this, because I'm more excited about having BBB on Cloudron and that isnt' even compatible with 18.04 yet!

And the worst is, they're explicitly putting all of the work in making BBB compatible with 18.04 first and they have no plans for 20.04.

[jdaviescoates]

@avatar1024 I'm not too bothered by that but yeah it's a bit mad they are still on 16.04

[ei8fdb]

@girish Absolutely. I wasn't suggesting to implement it "now-now". More the case of suggesting to gather data earlier (could be done at any time really), so that when it came to thinking about implementation, you had some data to base decisions on.

To get enough data I'd suggest running the research for ~ 2-3 weeks. Analysis would be much quicker ~days. I'll happy to help with both.

[girish]

Another brief update:

• We added auth wall functionality that apps without any notion of login can use
• i18n is mostly done, just lots of testing before we let translators do their work. @nebulon can give some updates on how the language is selected etc.
• We added httpPaths functionality so that @MooCloud_Matt can work on his OLS app. Also, see https://forum.cloudron.io/topic/3755/apache-ols-and-nginx-custom-benchmarks/5
• Reworked a lot of the internal dns stuff, so that we can have apps as dns servers (ad blocker apps)
• solr fts search was merged today.

Looks like we are set for a release next week!

[doodlemania2]

@girish Did the TURN server API auth stuff make it in for Nextcloud high performance backend'ness?

[nebulon]

@doodlemania2 unfortunately not, in fact I forgot about it

I have put it now into milestone for 6.1 so this does not happen again. Sorry about that.

[jdaviescoates]

@girish said in What's coming in 6.0 (take 2):

solr fts search was merged today.

This is great... but I'm intrigued to how exactly we'll be able to take advantage of it e.g. to search email/ files?

Will Cloudron Roundcube/ Rainloop/ Sogo, Nextcloud Mail somehow automagically take advantage of solr? or what?

I guess the same applies to Nextcloud files too - will search files on there somehow benefit from the power of solr?

Thanks!