Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.


  • Categories
  • Recent
  • Tags
  • Popular
  • Bookmarks
Skins
  • Light
  • Cerulean
  • Cosmo
  • Flatly
  • Journal
  • Litera
  • Lumen
  • Lux
  • Materia
  • Minty
  • Morph
  • Pulse
  • Sandstone
  • Simplex
  • Sketchy
  • Spacelab
  • United
  • Yeti
  • Zephyr
  • Dark
  • Cyborg
  • Darkly
  • Quartz
  • Slate
  • Solar
  • Superhero
  • Vapor

  • Default (No Skin)
  • No Skin
Collapse

Cloudron Forum

Apps | Demo | Docs | Install

Deploying apps that treat code as data

Scheduled Pinned Locked Moved Discuss
17 Posts 8 Posters 477 Views
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • girishG Offline
    girishG Offline
    girish Staff
    wrote on last edited by girish
    #1

    The problem is this: upstream authors often don't think of deployment and many times treat "code" as data. This means they write plugin and module infrastructure which writes all over the codebase. For the sake of this discussion, let's assume that we cannot convince the upstream authors that their approach is not good 😉

    Cloudron was designed with the idea of reproducible and immutable deployments. One of the biggest things is that it marks code as read only. This makes sure that apps can't write all over and that backups are actually correct. This is also needed for updates and restore to work reliably/smoothly. I have to say this works very well for most of the apps and yet use cases come up now and then which makes us wonder if we should let go of some of these rigid requirements.

    Here's some situations:

    • WordPress plugins want to write all over. Some caching plugins even write to the root wordpress directory, no idea why.
    • Dolibarr/odoo are like frameworks and you have to deploy a whole bunch of plugins to make them functional. Unfortunately, these plugins also use relative paths like include "../foo" which tends to break when we symlink mutable paths.
    • NodeBB plugins are node_modules. Latest versions of node don't allow symlinking node_modules anymore. This means we possibly have to put the entire code in some writable directory.
    • Koel wants to write to public directory to storage cover art. This is also the folder that contains all the js/css/webpack output! It also has code accessing things via relative paths. So, symlinking public to writable portion doesn't work.
    • Etherpad creates a file inside every node_module plugin directory.
    • Nextcloud plugins sometimes access relative paths as well.

    (On a separate note: none of the apps have a proper sandboxing and/or versioning mechanism for plugins. this means if a plugin breaks, nothing works.)

    Any ideas welcome on how such apps are maintained and updated outside of Cloudron.

    mehdiM timconsidineT 2 Replies Last reply
    9
  • girishG girish referenced this topic on
  • girishG girish referenced this topic on
  • mehdiM Offline
    mehdiM Offline
    mehdi App Dev
    replied to girish on last edited by
    #2

    In short, I am quite against letting go of these restrictions. I think their benefit is huge. I know it makes it a pain to package some apps, but I still think it's worth it.

    As for more specific points:

    @girish said in Deploying apps that treat code as data:

    Latest versions of node don't allow symlinking node_modules anymore

    Do you have a source on this ? I am very surprised, as a lot of nodejs utilities use symlinks in node_modules, things like Lerna (https://lerna.js.org/), or even native npm link.

    @girish said in Deploying apps that treat code as data:

    Koel wants to write to public directory to storage cover art. This is also the folder that contains all the js/css/webpack output! It also has code accessing things via relative paths. So, symlinking public to writable portion doesn't work.

    You could do it the other way around :

    • in Dockerfile, rename de folder /app/code/public2
    • create a symlink from /app/code/public to /app/data/public
    • in start.sh create /app/data/public, and inside it create symlinks to everything in /app/code/public2

    A bit hacky, but it should work I think 😄

    murgeroM girishG 2 Replies Last reply
    5
  • murgeroM Offline
    murgeroM Offline
    murgero App Dev
    replied to mehdi on last edited by
    #3

    @mehdi Reverse the symlink to make it work but otherwise, this is what I use to package certain apps as well

    --
    https://urgero.org
    ~ Professional Nerd. Freelance Programmer. ~
    Matrix: @murgero:urgero.org

    mehdiM 1 Reply Last reply
    2
  • mehdiM Offline
    mehdiM Offline
    mehdi App Dev
    replied to murgero on last edited by mehdi
    #4

    @murgero Are you sure ? I have seen (and made) some apps with a simple symlink, but I have never seen this sort of double-symlink (and no, i don't think there's anything to reverse in my explanation)

    robiR 1 Reply Last reply
    0
  • robiR Offline
    robiR Offline
    robi
    replied to mehdi on last edited by
    #5

    @mehdi I think he means in ln -s syntax it's reversed.

    @murgero is that right?

    Life of sky tech

    murgeroM 1 Reply Last reply
    1
  • robiR Offline
    robiR Offline
    robi
    wrote on last edited by robi
    #6

    @girish It may also be useful to have this concept available as a Tut/Writeup in the docs which you can link/send to upstream app devs to think about their app arch and adjust it for x, y, z benefits and easier inclusion into the Cloudron ecosystem.

    If they ignore it, we have workarounds.
    If they take it and make it, we all win 🏁

    Life of sky tech

    infogulchI 1 Reply Last reply
    2
  • infogulchI Offline
    infogulchI Offline
    infogulch
    replied to robi on last edited by infogulch
    #7

    My initial thought was along the lines of @robi's latest comment: as a first step lets have some good posts espousing the benefits of RW^X in app container directories and describe the pitfalls of letting everything devolve into a giant mess.

    Ideally the first thing you do when you find an app that treats its code frivolously is link the authors to posts that describe why what they're doing is a Bad Idea (tm) and how the problem they're trying to solve with this crazy code-as-data strategy can be solved more sanely. (Obviously you'll want to be somewhat more respectful than this when opening a random issue on someone's free project lol, but really guys... don't do this.)


    As far as the initial question where we attempt to proceed assuming we can't tame the app's behavior, maybe a "messy mode" for apps where backups include more directories than just /data. Backups would be much larger. Images maybe couldn't be based on the cloudron base image. Updates would become very messy (maybe just overwrite files with ones from the new image?). At that point I'm not sure if it would be sustainable to officially support such apps at all, they might require a white glove support for individual installations.

    1 Reply Last reply
    1
  • girishG Offline
    girishG Offline
    girish Staff
    replied to mehdi on last edited by
    #8

    @mehdi said in Deploying apps that treat code as data:

    @girish said in Deploying apps that treat code as data:

    Latest versions of node don't allow symlinking node_modules anymore

    Do you have a source on this ? I am very surprised, as a lot of nodejs utilities use symlinks in node_modules, things like Lerna (https://lerna.js.org/), or even native npm link.

    Yes, see https://github.com/npm/cli/issues/3669 and (related) https://github.com/nodejs/node/issues/22922 . We symlink node_modules in the nodebb app which doesn't work anymore because of this. Not 100% sure but I think npm link creates symlinks inside node_modules which is different from this one where node_modules itself is symlinked.

    MooCloud_MattM 1 Reply Last reply
    2
  • P Offline
    P Offline
    privsec
    wrote on last edited by
    #9

    Why not offer an option within the app to determine whether or not it is write protected?

    infogulchI 1 Reply Last reply
    2
  • infogulchI Offline
    infogulchI Offline
    infogulch
    replied to privsec on last edited by
    #10

    @privsec If the user installs a plugin which is stored in the code area, and the backup only backs up the data area, when the app is deleted and restored from a backup it will lose the plugin and potentially also data.

    P 1 Reply Last reply
    0
  • P Offline
    P Offline
    privsec
    replied to infogulch on last edited by
    #11

    @infogulch
    could the backup locations be updated?

    1 Reply Last reply
    0
  • murgeroM Offline
    murgeroM Offline
    murgero App Dev
    replied to robi on last edited by
    #12

    @robi Ah yes - this is what I was thinking about.

    @mehdi - My apologies, I may have been thinking of alternate syntax to the symlinking command.

    --
    https://urgero.org
    ~ Professional Nerd. Freelance Programmer. ~
    Matrix: @murgero:urgero.org

    1 Reply Last reply
    3
  • MooCloud_MattM Offline
    MooCloud_MattM Offline
    MooCloud_Matt
    replied to girish on last edited by
    #13

    @girish

    Probably a possible solution is to allow multiple Volumes to be mounted to the container, so there is no need for a syslink, and is still easy to backup, the backup script knows which volume to backup for each container.
    Is a bit more work on the Backup script, but should allow an easier port to Cloudron.

    Or, I see as an alternative the release of the Manifest standard as a separate Spec from Cloudron, like the "Compose Spec".
    This should allow developers to better understand what cloudron and the manifest support and how they work, and hopefully start to act more in line with safer containerization standards.

    Matteo. R.
    Founder and Tech-Support Manager.
    MooCloud MSP
    Swiss Managed Service Provider

    robiR 1 Reply Last reply
    1
  • timconsidineT Offline
    timconsidineT Offline
    timconsidine App Dev
    replied to girish on last edited by timconsidine
    #14

    @girish : it will for sure be impossible to persuade others to do it the Cloudron way, so the problem has to be addressed.

    Personally I feel the Cloudron stability and reliability is SO IMPORTANT that it must be preserved and it is the priority. If it is impossible to find a solution, we have to accept a situation like "sorry, this XXX app is such a mess, it is not possible to make it comply with Cloudron standards, and even though it is good functionality, and we could possibly package it, we're not going to. Because it's a mess."

    While I don't have a technical solution approach to the problem, I think it's almost inevitable that we end up with another status of apps :

    • OFFICIAL (package tested, released, maintained, conforms to Cloudron standards)
    • UNSTABLE (although I don't like this word : should be Beta or not-yet-officially-released)
    • MESS or "Pig's Breakfast" (packaged, works, use it, but maintained as best endeavours, esp ref backups, because upstream dev does not conform to Cloudron standards)

    Not ideal, but better than lowering Cloudron standards. Would rather accept not having XXX app in the store.

    1 Reply Last reply
    11
  • mehdiM Offline
    mehdiM Offline
    mehdi App Dev
    wrote on last edited by
    #15

    I quite agree with @timconsidine . We could have these apps on Cloudron, but with a big fat red warning that says the app is unsupported :

    • zero user support
    • larger backups
    • no auto-update

    If unhappy: bother the original author.

    1 Reply Last reply
    8
  • robiR Offline
    robiR Offline
    robi
    replied to MooCloud_Matt on last edited by
    #16

    @moocloud_matt said in Deploying apps that treat code as data:

    a possible solution is to allow multiple Volumes to be mounted to the container

    This is along the lines of OpenEBS storage discussion here.

    Elegant way to install an "elevator" to any directory in the container and swap out files we might need, without disrupting the app.

    Life of sky tech

    MooCloud_MattM 1 Reply Last reply
    0
  • MooCloud_MattM Offline
    MooCloud_MattM Offline
    MooCloud_Matt
    replied to robi on last edited by MooCloud_Matt
    #17

    @robi
    in general, having support for multiple drivers for Volumes is a good idea, but set in the Manifest standard the support for multiple volumes, could solve the issue at least partially.

    Matteo. R.
    Founder and Tech-Support Manager.
    MooCloud MSP
    Swiss Managed Service Provider

    1 Reply Last reply
    1

  • Login

  • Don't have an account? Register

  • Login or register to search.
  • First post
    Last post
0
  • Categories
  • Recent
  • Tags
  • Popular
  • Bookmarks
  • Login

  • Don't have an account? Register

  • Login or register to search.