What's coming in 6.0 (take 2)

[girish]

Some updates on volume management. There is now a volume view.

You add a host path as a volume and give it a name. This is most likely some EBS/external hard disk/Block Storage. There is also the file manager integration (the button next to the delete button).

You can then go to the app view and mount these volumes into the apps. There is a new Storage section.

There is a section overload in the app configure view We are looking to reducing the list of sections in the left pane. It's making things hard to find.

[atridad]

This looks wonderful!

Off the top of my head, I could see the Console + Repair sections being merged and the Resources + Access Control + Location sections being merged. But honestly I am struggling to come up with names so I suppose thats why you have them the way you have them now.

[marcusquinn]

@girish Amazing!

It's almost like you develop with the power of telepathy sometimes

[marcusquinn]

@girish Any chance of you adding a button in for resizing the partition after upgrading a VM?

[girish]

@marcusquinn Not for this release but hope to implement more volume management tools (like mounting nfs/sshfs, backing up volumes, resizing etc) as we go.

[Lonkle]

Not a feature I was expecting but I know I'll take advantage of it. This is pretty cool!

[jdaviescoates]

@marcusquinn said in What's coming in 6.0 (take 2):

@girish Any chance of you adding a button in for resizing the partition after upgrading a VM?

+1

Resizing my backup volume on Hetzner is one of the few reasons I have to occasionally ssh into my VPS.

[robi]

Externalizing a volume to a Minio/S3/Object store would be nice.

[MooCloud_Matt]

@robi
s3 filesystem are not so mature, i read a lot on them because use s3 with dovecot in our mail server and the performance are terrible in u use s3 as FS, 20/40 ms depending on where you host the s3 server.

The optimal solution is to use plugin or software that integrate s3 in to the App/script that you want to use.

[mehdi]

@girish I could not have imagined a better system ! Looks amazing !

Three questions:

1/ How do you decide where in the app these are gonna be mounted? Is it in the same path for all apps ? Or could I create a given volume, and mount it at /app/data/toto in app 1 and /app/data/tata in app 2 ?

2/ How does it work when there already is data in the app's original /app/data before you mount the volume to it? Is it just hidden as long as the volume is mounted?

3/ How does it work with regards to backups?

[nebulon]

For 1 and 2: Those volumes will be mounted into the apps based on /media/<volumename> and thus one volume associated with apps will always have the same mountpoint within the apps.

For 3: Initially those won't be backed up at all, but we will likely add optional backup per volume as well. The user has to be aware of the fact that volumes are not part of the app backups as such!

[necrevistonnezr]

@nebulon said in What's coming in 6.0 (take 2):

For 1 and 2: Those volumes will be mounted into the apps based on /media/<volumename> and thus one volume associated with apps will always have the same mountpoint within the apps.

For 3: Initially those won't be backed up at all, but we will likely add optional backup per volume as well. The user has to be aware of the fact that volumes are not part of the app backups as such!

That’s the difference to storage mounted in „Resources“ - this will get backed up, correct?

[nebulon]

The volumes are setup outside of the apps first and are essentially independent. Once you've created a volume you can then mount that into an app via the Storage section in the app configuration. Even if mounted though, it will not be part of the app backup, since the main reason for volumes is to be mounted into multiple apps. For that we have to first figure out how to manage those with regards to backups.

[d19dotca]

@nebulon That makes sense. I’m of the opinion that optional block storage should effectively be backed up at the host level not the application level. In other words, taking a snapshot or something like that which many providers allow even volumes / block storage to be snapshotted. Much like how we need to snapshot our actual VPS on occasion too for a backup strategy of the entire server.

[necrevistonnezr]

@nebulon said in What's coming in 6.0 (take 2):

The volumes are setup outside of the apps first and are essentially independent. Once you've created a volume you can then mount that into an app via the Storage section in the app configuration. Even if mounted though, it will not be part of the app backup, since the main reason for volumes is to be mounted into multiple apps. For that we have to first figure out how to manage those with regards to backups.

No, I mean external volumes that you can currently already move the data dir to under „Resources“ in the app settings - this gets backed up, correct?

(

[girish]

We haven't decided on the naming yet, but maybe Volumes can be instead renamed to External Media or something, if it clarifies things. Open to other suggestions as well.

[robi]

@MooCloud_Matt this is only true if you use dumb S3 connectors that treat it like a filesystem instead of an object store.

It's a very different experience when things are treated properly as objects.

[iqweb]

Re: Volumes - I think this will create a solution for something that I have been trying to resolve - i.e. Nextcloud app backups WITHOUT the data!

I want to be able to back up the app via Cloudron natively - but backup the data via borg/restic/kopia or the like. This looks like it could resolve this issue if the data was an attached volume. Very much looking forward to this feature if it's like that.

[girish]

@iqweb said in What's coming in 6.0 (take 2):

Re: Volumes - I think this will create a solution for something that I have been trying to resolve - i.e. Nextcloud app backups WITHOUT the data!

Yes, I think this will solve that use case. You mount a volume into nextcloud and configure nextcloud to use it as external storage.

[imc67]

@girish can you also mount an ‘on server’ folder like that?

[girish]

@imc67 Yes. Currently, the main restriction is that you can only add host paths under /mnt and /media as a volume. So, you can create some NFS mount or SSHFS mount there and then add it as a volume and then mount it into an app.

[scooke]

@ruihildt There is a free tier, and if the "5 Mailboxes" means there is some functional email, then that is not bad.

[ruihildt]

@scooke I'm not sure what you're replying to?

[jdaviescoates]

@girish can't wait to have volumes to play with! what's your guestimate ETA?

[girish]

@jdaviescoates Cloudron 6 is ~2-3 weeks away. The translation project is quite a massive change, so maybe I am being overlay optimistic here.

[jdaviescoates]

@girish all power to your elbow! (i.e. good luck!)

[scooke]

@ruihildt Your comment from 26 days ago, "As an aside, they are advertising Cloudron as free software, which is not correct at the moment." I came across it as I read through the entire thread.

[jdaviescoates]

@scooke said in What's coming in 6.0 (take 2):

@ruihildt Your comment from 26 days ago, "As an aside, they are advertising Cloudron as free software, which is not correct at the moment." I came across it as I read through the entire thread.

I think you've missed the point

@ruihildt was talking about free as in freedom (which is how that project advertises Cloudron), not free as in price.

[scooke]

@jdaviescoates I figured so. But I wonder why @ruihildt bothered to point out what he did. The few lines I found on the website aren't actually advertising Cloudron as %100 open source, but rather that the code they use is open source. I'd be more bothered by their approach, like Framasoft, where if the user has no idea about the existence of Cloudron, in this case, the company's description of what they do (https://osinum.fr/) makes it sound like it is and has been all their own doing. Much of what they promise to do is based on whether Cloudron, or all the other myriad open source projects in use, has or will implement it, but they don't clearly specify that. If I found them, and started using them, and THEN discovered they actually use Cloudron, I'd be a little miffed, and would just switch over to Cloudron.
Anyway, I have no clue what "being sponsored by Medias-Cite" means, but if the Cloudron team are fine with it, and this "sponsoring" helps move Cloudron forward, great.

(A bit more reading shows that they, osinum.fr, are charging a minimum 150 EURO/month!!! I guess they are paying Cloudron for the Premium Plus plan. They offer, for almost triple the price of Cloudron's pricing,
1 domain name (.fr, .com, .org, .net)
Accommodation in France
Shared server
100 GB
5 applications
50 users
Management of user groups
Daily backups
Surveillance / Monitoring / Security
Email support

Maybe I should get into offering cloudron!!

[girish]

hey guys, let's stick to Cloudron 6 related discussion here Feel free to open new thread for other topics.

[jdaviescoates]

@girish fair enough

Just to quickly say (sorry) to @scooke re:

the user has no idea about the existence of Cloudron

I'm not sure where you are looking but on https://osinum.fr/ there is this:

Which translates as:

"The core of OSINUM is the Cloudron software and its team of developers."

So 1) they seem quite clear about being based on Cloudron, and 2) they've actually changed what it used to say which @ruihildt was commenting on.

Previously it said: "The core of OSINUM is the open source software Cloudron and its team of developers."

Edit: see my screenshot here https://twitter.com/jdaviescoates/status/1314243996441620480

/end.

[scooke]

@jdaviescoates @girish D'accord.

[jdaviescoates]

@girish said in What's coming in 6.0 (take 2):

You add a host path as a volume and give it a name. This is most likely some EBS/external hard disk/Block Storage.

I'm assuming volumes will support CIFS unlike current external data storage, right? (if not I had better cancel the Hetzner storage box I just ordered for this purpuse (or just start using it for backups where CIFS is already allowed)! )

[imc67]

@girish said in What's coming in 6.0 (take 2):

@imc67 Yes. Currently, the main restriction is that you can only add host paths under /mnt and /media as a volume. So, you can create some NFS mount or SSHFS mount there and then add it as a volume and then mount it into an app.

I really would appreciate it if you could create a step by step instruction in the docs by that time. I guess more users with large base “disk” would want to use this!

[mehdi]

@jdaviescoates said in What's coming in 6.0 (take 2):

I'm assuming volumes will support CIFS unlike current external data storage, right? (if not I had better cancel the Hetzner storage box I just ordered for this purpuse (or just start using it for backups where CIFS is already allowed)! )

The underlying problem is still there, CIFS / SMB don't support users and groups in the FS, so i guess it will depend on the apps, if they rely on FS permissions or not

[jdaviescoates]

@mehdi hmz, thanks, so I'm guessing likely not then.

So Hetzner storage boxes likely a no no for volumes (I guess I'll cancel the one I've just ordered, or just use it for backups)

I wonder what would be a good option...

I guess maybe Hetzner Volumes would work well, but they cost a lot more (a 500GB storage box is 5.68€/mo, whereas a 500GB Volume is 24€/mo)

[girish]

@jdaviescoates said in What's coming in 6.0 (take 2):

So Hetzner storage boxes likely a no no for volumes (I guess I'll cancel the one I've just ordered, or just use it for backups)

So, there's two kinds of storages:

• Data Directory - This is a location used by the app and only the app uses it exclusively. The app will store say attachments, profile pics, avatars, plugin code and whatever it needs in this directory. Data directory has to be ext4 because the packaging code requires file system permissions to work etc. This is the location you specify in https://docs.cloudron.io/storage/#app-data-directory . You can think of this app as part of an app's "state" (just like the database) and it is thus backed up and restored.

• External Directory/Volumes - This location can be any file system. Mostly the app just reads file (docs, music, movies etc) from this. So the permissions don't matter. It is also expected that these directories can be shared across apps. This is similar to the home directory on linux or My Documents on Windows. It's just separate directory that apps have access to. If you uninstall, backup apps, that folder is untouched. For these kind of directories, the file system type is not relevant. It can be anything - sshfs, cifs, ext2 whatever.

[jdaviescoates]

@girish OK, thanks, that's clearer.

However, some of the music streaming apps etc like Navidrome (which is what I thought I'd use a Hetzner Storage box CIFS for) can have users with different permissions too, so still not sure if that'll be possible?

Although the way I'd actually want to use it is that only one person (me) will be able to upload music, and all other users will just have read access, so I guess that could work fine?

[girish]

@jdaviescoates said in What's coming in 6.0 (take 2):

However, some of the music streaming apps etc like Navidrome (which is what I thought I'd use a Hetzner Storage box CIFS for) can have users with different permissions too, so still not sure if that'll be possible?

Yes, definitely possible. App users are entirely different from file system permissions. Without getting too technical, users inside an app are just "virtual users". They just exist inside the app's database.

[jdaviescoates]

@girish OK!

So maybe I will start uploading my 290GB of music up to my new 500GB Hetzner Storage Box to use like this as soon as 6.0 goes live!

[jdaviescoates]

@girish said in What's coming in 6.0 (take 2):

see the WIP tag

Oooh, just looked at that for the first time in a while and I see BigBlueButton and Jitsi Meet are included... who is actively working on them atm, is it you @girish @nebulon ?

Cloudron 6.0 with full text email search, volumes and BBB/ Jitsi would be SO awesome! (loving all the new music/ media apps too!)

[mehdi]

Once there are proper Volumes, my River app won't have too much reason to exist, so you can expect a few other apps soon after 6.0 release as I split river into independent apps (namely SickChill, CouchPotato, and Transmission)

[Lonkle]

@mehdi said in What's coming in 6.0 (take 2):

Once there are proper Volumes, my River app won't have too much reason to exist, so you can expect a few other apps soon after 6.0 release as I split river into independent apps (namely SickChill, CouchPotato, and Transmission)

Did your River app combine those apps with an added external storage connection bit?

[mehdi]

@lonk It combines Jellyfin (for which there's already an app now), Transmission, SickChill, Couchpotato (the 3 I just mentioned), a custom file manager (built before Cloudron had one), and a custom TV Shows and Movies streaming interface (built before I integrated Jellyfin). Plus a few custom things, like a script to auto-remove finished torrents on transmission, and a script to auto-convert videos to MP4 for easy streaming in the browser (for my custom streaming interface. I disabled it now that I mainly use jellyfin, which handles this automatically).

[girish]

@mehdi said in What's coming in 6.0 (take 2):

Once there are proper Volumes, my River app won't have too much reason to exist, so you can expect a few other apps soon after 6.0 release as I split river into independent apps (namely SickChill, CouchPotato, and Transmission)

When you have time, please split them out and I can start approving them as unstable already. It also gives me a good test bed to test the volumes stuff across apps and permissions.

[mehdi]

@girish said in What's coming in 6.0 (take 2):

When you have time, please split them out and I can start approving them as unstable already.

OK, will do. I will have to pick your brain a little about how to handle authentication for these apps. In river, they're behind a custom auth proxy, as they don't handle auth by themselves.

[girish]

@mehdi said in What's coming in 6.0 (take 2):

OK, will do. I will have to pick your brain a little about how to handle authentication for these apps.

Would it make sense to bring that into box code somehow? Maybe we add some flag in the manifest to turn on this "authentication wall".

@nebulon Is it possible to have a login screen like the surfer app but served from box code? I guess session management possibly won't work since app has no clue about this? Maybe logout won't work as well since there is no logout button in the app.

[mehdi]

@girish said in What's coming in 6.0 (take 2):

Would it make sense to bring that into box code somehow? Maybe we add some flag in the manifest to turn on this "authentication wall".

It would be possible yeah. An other possibility would be to have a standard way to do it in the "base image", to make it easy to implement in the apps that need it. I would probably go with the "base image" path, because it would be easier to configure on an app level IMO.

[girish]

@mehdi Good idea. Maybe then for the moment we can copy/paste apache configs in every app (like https://git.cloudron.io/cloudron/simple-torrent-app/-/blob/master/apache/cloud-torrent.conf ) for now and once we have a something common, I can try to put it in the next base image.

[Lonkle]

I really like that approach as well. I'll follow what you guys create and what changes occur to the base image! ️

[nebulon]

So the idea is to put some kind of "framework" into the base image, which can be used by apps? Wouldn't that anyways still not mean that an app needs to be patched for at least the logout action? Also would we do this as a php set of features? I do like to not pull this into the platform code as such, as that does not increase dependency on that.

Alternately, we could certainly add a login screen served up with some kind of session. The question then, as already mentioned, is how to logout. We could provide the app with a logout link, still that needs patching the app to some extent.

[mehdi]

@nebulon IMO the login part is much more important than the logout part. We can even do completely without the logout at all in the interface, with just a /logout URL that one would have to enter manually (if ever).

As to the precise tech to use, I already have a working version in Node.JS in river that I could isolate. If you guys prefer to re-do it in PHP instead or something else, it's your choice.

[nebulon]

Right, I was only bringing up php since that might be more commonly already be used within such apps, I would prefer a nodejs solution though. Maybe we can collect some arguments for and against adding to the base image or into the platform.

If we add it to the platform, we could have it more easily streamlined with the Cloudron look and feel, however within the base image the app could style it more towards its own look and feel. Putting it in the platform on the other hand would allow support translation now once it fully landed. Also say we use a nodejs based version, then we have to keep running an additional process with possibly another proxy even?

[jimcavoli]

Just pointing out that this might be better served by using something like a proper gateway and load balancing solution in front of the apps like Kong rather than NGINx on the box. An add-on for a basic auth screen could just be a config tweak to the box ingress which hits an auth wall - added advantage would be much more flexible routing to apps (sub-path, etc.) and flexible options like having multiple DNS names resolving to the straight A record for things like apps that can serve multiple domain names off one instance.

[robi]

Bringing in some service mesh concepts would be interesting.

[MooCloud_Matt]

@jimcavoli
You mean to use Kong or Traefik (this are exeple) as incoming proxy for every container ?

If yes, we need to consider how reliable is Nginx and the feature that can be easily added (Proxy_Cache, brotli, WAF, ...)

[jimcavoli]

@moocloud_matt yeah, there's currently a box level NGINx proxy - my idea is to replace that with a very hand-wavy something else which may be capable of shimming authentication for those things that just don't have it inbuilt (Kong) or if we go a different route on that sort of thing, we could use Traefik or similar at that layer. I think the use cases are intriguing enough to at least try the Kong route and fall back to something like Traefik if need be

[MooCloud_Matt]

@jimcavoli
for adding .htpasswd support ? or what kind of auth are you talking about?

[jimcavoli]

@moocloud_matt Kong is a pretty expansive topic on its own, but the idea would be with it in charge of ingress/routing, you could have a simple add-on config that would take care of enabling a plugin like https://docs.konghq.com/hub/kong-inc/ldap-auth/ on the route to a particular app, so you could have HTTP basic auth, but completely backed by the full Cloudron user store for any app that doesn't have its own scheme, providing clean headers that could be easily handled by that app's eb server or whatever

[girish]

Kong is indeed a separate complex topic. I think for the moment, if we had some template that people can quickly copy over to the app to get auth screen/login, it will help already. @nebulon do we have such a template already ? (like the one we use for our internal apps).