Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.


  • Categories
  • Recent
  • Tags
  • Popular
  • Bookmarks
Skins
  • Light
  • Cerulean
  • Cosmo
  • Flatly
  • Journal
  • Litera
  • Lumen
  • Lux
  • Materia
  • Minty
  • Morph
  • Pulse
  • Sandstone
  • Simplex
  • Sketchy
  • Spacelab
  • United
  • Yeti
  • Zephyr
  • Dark
  • Cyborg
  • Darkly
  • Quartz
  • Slate
  • Solar
  • Superhero
  • Vapor

  • Default (No Skin)
  • No Skin
Collapse

Cloudron Forum

Apps | Demo | Docs | Install

Using Object Storage for Cloudron Docker and Cloudron Backup?

Scheduled Pinned Locked Moved Discuss
cloudronobject storagedockerbackup-storagestorage
20 Posts 6 Posters 206 Views
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • L Offline
    L Offline
    LoudLemur
    wrote on last edited by
    #1

    These two are quite large and between them consume a large proportion of a Cloudron instance's storage. Large files lend themselves to object storage. Would it be a good idea and relatively simple to have Cloudron keep these two components on some dedicated, object storage? This would free up space for other uses on the instance.

    1 Reply Last reply
    0
  • girishG Offline
    girishG Offline
    girish Staff
    wrote on last edited by
    #2

    Docker has to be kept on disk, it doesn't support object storage. As for, Cloudron backup, you can configure this to be on any S3 compatible provider.

    robiR 1 Reply Last reply
    1
  • robiR Offline
    robiR Offline
    robi
    replied to girish on last edited by
    #3

    @girish I think he meant docker images, not docker itself.

    Life of sky tech

    1 Reply Last reply
    1
  • nebulonN Offline
    nebulonN Offline
    nebulon Staff
    wrote on last edited by
    #4

    If you were referring to docker images, then those are not even backed up.

    L robiR 2 Replies Last reply
    0
  • L Offline
    L Offline
    LoudLemur
    replied to nebulon on last edited by LoudLemur
    #5

    @nebulon Thanks for looking at this. The two large items I was looking at are the first two that appear if you go into your Cloudron's "System Information" and look at the Disk Usage chart at the top right. You can probably see that the first two components are quite large, relative to everything else. They are named "cloudron-backup" and "docker".

    By the way, any progress on supporting the deletion of particular backup files?

    1 Reply Last reply
    0
  • robiR Offline
    robiR Offline
    robi
    replied to nebulon on last edited by
    #6

    @nebulon it's more about keeping those large docker images on s3 off the local disk as they just consume lots of space. Not about backup.

    Life of sky tech

    girishG 1 Reply Last reply
    1
  • girishG Offline
    girishG Offline
    girish Staff
    replied to robi on last edited by
    #7

    @robi sorry, I meant docker images in my earlier post. Currently, docker cannot run apps with images external to the server. They have to be downloaded locally to the server.

    robiR 1 Reply Last reply
    1
  • robiR Offline
    robiR Offline
    robi
    replied to girish on last edited by
    #8

    @girish Right, unless that is a local mount of a remote FS or s3 object store via rclone.

    It would be beneficial to not have to waste so much space from the OS as well as docker images plus apps data all locally.

    This also brings up the issue if the monolithic large base image vs small image layers of just what is needed for each app.

    You wouldn't have to republish all apps with a new base image, right?

    Life of sky tech

    scookeS 1 Reply Last reply
    0
  • scookeS Offline
    scookeS Offline
    scooke
    replied to robi on last edited by
    #9

    @robi said in Using Object Storage for Cloudron Docker and Cloudron Backup?:

    This also brings up the issue if the monolithic large base image vs small image layers of just what is needed for each app.

    If by this you mean Cloudron could have one MySQL image for all apps needing that, or one NGINX image for all apps needing proxying, etc. ... I'm not such a fan of that idea. I've tried that with what, portainer, and so much hassle. Even though it requires more room, I much prefer Cloudron putting everything an app needs into one image.

    A life lived in fear is a life half-lived

    MooCloud_MattM 2 Replies Last reply
    0
  • MooCloud_MattM Offline
    MooCloud_MattM Offline
    MooCloud_Matt
    replied to scooke on last edited by
    #10

    Hello @scooke
    I will replay splitting your question/douth a part.

    • Cloudron already use just one container for all the MySQL database.
      What @robi is suggesting is to optimize how the docker image is created.
    • A monolithic or layer image can obtain the same, and it doesn't matter what control panel you use.
    • images built with layers will be a lot more space efficient because they share the same basic "image from" and they will not use space that is not requested.
      example: if you just use MySQL for a WordPress, why should you have MongoDB, pSQL, turn on your images?
      But if you add all the images that you would create using layers together they will use the same space as the current monolithic image approach.

    Matteo. R.
    Founder and Tech-Support Manager.
    MooCloud MSP
    Swiss Managed Service Provider

    1 Reply Last reply
    1
  • MooCloud_MattM Offline
    MooCloud_MattM Offline
    MooCloud_Matt
    replied to scooke on last edited by
    #11

    @scooke said in Using Object Storage for Cloudron Docker and Cloudron Backup?:

    I much prefer Cloudron putting everything an app needs into one image.

    This is not advised because you will not need everything every time, which will use your space, increasing security issues, because, after all, it's harder to realize updates into a monolithic base image.

    Matteo. R.
    Founder and Tech-Support Manager.
    MooCloud MSP
    Swiss Managed Service Provider

    1 Reply Last reply
    1
  • MooCloud_MattM Offline
    MooCloud_MattM Offline
    MooCloud_Matt
    wrote on last edited by MooCloud_Matt
    #12

    This is an example if a server was hosting just a single WordPress, if we can optimize the same cloudron base image, this is how many unused dependencies and packages are included that we don't use.

    With just WordPress, we have to have the light base image + some debug tools + MySQL and the wordpress image itself that is base on apache + PHP (just the selected version that you are interested in) + wordpress core.

    This will reduce by a reasonable amount of the space used by the cloudron base image.

    99ba6ab6-50f6-4a5a-8ccf-205cc5ecb5af-image.png

    And to conclude, I think that you can't store in an easy and reliable way images on an s3 server, because images are stored just once, even if you start multiple containers.

    Matteo. R.
    Founder and Tech-Support Manager.
    MooCloud MSP
    Swiss Managed Service Provider

    scookeS robiR 2 Replies Last reply
    1
  • scookeS Offline
    scookeS Offline
    scooke
    replied to MooCloud_Matt on last edited by
    #13

    @MooCloud_Matt I like the diagram. Did you make it?
    Also, isn't this exactly what Cloudron does though?

    A life lived in fear is a life half-lived

    MooCloud_MattM 1 Reply Last reply
    0
  • MooCloud_MattM Offline
    MooCloud_MattM Offline
    MooCloud_Matt
    replied to scooke on last edited by
    #14

    @scooke said in Using Object Storage for Cloudron Docker and Cloudron Backup?:

    Did you make it?

    yes, I did =D

    Also, isn't this exactly what Cloudron does though?

    Nope, its not, cloudron includes everything, you always have:
    Node 18.12.1
    Go 1.19.3
    Ruby 3.0
    PHP 8.1
    Python 3.10
    Redis
    Mysql
    Mongo
    pSQL

    This for version 4.
    If you want to keep compatibility with an older WordPress version because your plugin will not support 8.1.
    you have to keep all it's included in cloudron base image v3

    Matteo. R.
    Founder and Tech-Support Manager.
    MooCloud MSP
    Swiss Managed Service Provider

    scookeS 1 Reply Last reply
    1
  • scookeS Offline
    scookeS Offline
    scooke
    replied to MooCloud_Matt on last edited by
    #15

    @MooCloud_Matt I didn't realize this! So my WP app, my Vaultwarden, Matrix, Mastodon, Github, Grav, Jellyfin, Navidrome, Nextcloud, Vikunja, Umami, OpenVPN, LAMP app, etc. has ALL of those in the image that is installed when I click Install X app in the app store?? Why are all of those needed? I have 52 apps installed... so each of those apps has ALL of that software in its image?? Wow.

    I assumed that, for example with the LAMP app, that the image would include Apache, MySQL, and PHP. No node, no Go, no Ruby, etc. This is quite eye-opening.

    A life lived in fear is a life half-lived

    MooCloud_MattM 1 Reply Last reply
    1
  • MooCloud_MattM Offline
    MooCloud_MattM Offline
    MooCloud_Matt
    replied to scooke on last edited by
    #16

    @scooke
    Not exactly; the image is stored once and used multiple times.
    1 image multiple containers.
    That means that you will not be impacted by a more optimized image because any way you use all its components.

    But it will impact those who rent a 10 or 20 gb VPS because they will most likely use just part of that image, maybe just wordpres, or just a small nextcloud.
    and they don't need ruby or node

    Matteo. R.
    Founder and Tech-Support Manager.
    MooCloud MSP
    Swiss Managed Service Provider

    1 Reply Last reply
    1
  • robiR Offline
    robiR Offline
    robi
    replied to MooCloud_Matt on last edited by
    #17

    @MooCloud_Matt said in Using Object Storage for Cloudron Docker and Cloudron Backup?:

    And to conclude, I think that you can't store in an easy and reliable way images on an s3 server, because images are stored just once, even if you start multiple containers.

    You can if you attach remote storage via things like rclone.

    Life of sky tech

    MooCloud_MattM 1 Reply Last reply
    0
  • MooCloud_MattM Offline
    MooCloud_MattM Offline
    MooCloud_Matt
    replied to robi on last edited by
    #18

    @robi
    yes, but then your image is used to run all your apps, which means that all those apps to run will have to get data from s3.

    Remember that that image is not replicated anywhere else in the Filesystem, the container read their dependency from that file.
    It would be like booting your server from s3, not a good idea, but that doesn't mean that storage footprint optimization is not possible in another way.

    Matteo. R.
    Founder and Tech-Support Manager.
    MooCloud MSP
    Swiss Managed Service Provider

    robiR 1 Reply Last reply
    0
  • robiR Offline
    robiR Offline
    robi
    replied to MooCloud_Matt on last edited by
    #19

    @MooCloud_Matt agreed, yet once it's loaded into RAM, there isn't much access to the filesystem at that layer. "Streaming" apps from S3 will have an impact on startup times and performance, but saves a lot of space.

    A hybrid mode could be where long term layers are cached in RAM, short term offloaded remotely and essential app data kept locally which CL backups take care of.

    Life of sky tech

    MooCloud_MattM 1 Reply Last reply
    0
  • MooCloud_MattM Offline
    MooCloud_MattM Offline
    MooCloud_Matt
    replied to robi on last edited by
    #20

    @robi
    On that, I don't have the data or experience to talk.
    But 100% sure you can't use wasabi latency, I think smb or better NFS should be a better option, but I'm not an expert.
    and Using so much ram for it, im not big fun about it, but I would have to try it before to have a real feedback

    Matteo. R.
    Founder and Tech-Support Manager.
    MooCloud MSP
    Swiss Managed Service Provider

    1 Reply Last reply
    0

  • Login

  • Don't have an account? Register

  • Login or register to search.
  • First post
    Last post
0
  • Categories
  • Recent
  • Tags
  • Popular
  • Bookmarks
  • Login

  • Don't have an account? Register

  • Login or register to search.