Deploying apps that treat code as data

[mehdi]

In short, I am quite against letting go of these restrictions. I think their benefit is huge. I know it makes it a pain to package some apps, but I still think it's worth it.

As for more specific points:

@girish said in Deploying apps that treat code as data:

Latest versions of node don't allow symlinking node_modules anymore

Do you have a source on this ? I am very surprised, as a lot of nodejs utilities use symlinks in node_modules, things like Lerna (https://lerna.js.org/), or even native npm link.

@girish said in Deploying apps that treat code as data:

Koel wants to write to public directory to storage cover art. This is also the folder that contains all the js/css/webpack output! It also has code accessing things via relative paths. So, symlinking public to writable portion doesn't work.

You could do it the other way around :

• in Dockerfile, rename de folder /app/code/public2
• create a symlink from /app/code/public to /app/data/public
• in start.sh create /app/data/public, and inside it create symlinks to everything in /app/code/public2

A bit hacky, but it should work I think

[murgero]

@mehdi Reverse the symlink to make it work but otherwise, this is what I use to package certain apps as well

[mehdi]

@murgero Are you sure ? I have seen (and made) some apps with a simple symlink, but I have never seen this sort of double-symlink (and no, i don't think there's anything to reverse in my explanation)

[robi]

@mehdi I think he means in ln -s syntax it's reversed.

@murgero is that right?

[robi]

@girish It may also be useful to have this concept available as a Tut/Writeup in the docs which you can link/send to upstream app devs to think about their app arch and adjust it for x, y, z benefits and easier inclusion into the Cloudron ecosystem.

If they ignore it, we have workarounds.
If they take it and make it, we all win

[infogulch]

My initial thought was along the lines of @robi's latest comment: as a first step lets have some good posts espousing the benefits of RW^X in app container directories and describe the pitfalls of letting everything devolve into a giant mess.

Ideally the first thing you do when you find an app that treats its code frivolously is link the authors to posts that describe why what they're doing is a Bad Idea (tm) and how the problem they're trying to solve with this crazy code-as-data strategy can be solved more sanely. (Obviously you'll want to be somewhat more respectful than this when opening a random issue on someone's free project lol, but really guys... don't do this.)

---

As far as the initial question where we attempt to proceed assuming we can't tame the app's behavior, maybe a "messy mode" for apps where backups include more directories than just /data. Backups would be much larger. Images maybe couldn't be based on the cloudron base image. Updates would become very messy (maybe just overwrite files with ones from the new image?). At that point I'm not sure if it would be sustainable to officially support such apps at all, they might require a white glove support for individual installations.

[girish]

@mehdi said in Deploying apps that treat code as data:

@girish said in Deploying apps that treat code as data:

Latest versions of node don't allow symlinking node_modules anymore

Do you have a source on this ? I am very surprised, as a lot of nodejs utilities use symlinks in node_modules, things like Lerna (https://lerna.js.org/), or even native npm link.

Yes, see https://github.com/npm/cli/issues/3669 and (related) https://github.com/nodejs/node/issues/22922 . We symlink node_modules in the nodebb app which doesn't work anymore because of this. Not 100% sure but I think npm link creates symlinks inside node_modules which is different from this one where node_modules itself is symlinked.

[privsec]

Why not offer an option within the app to determine whether or not it is write protected?

[infogulch]

@privsec If the user installs a plugin which is stored in the code area, and the backup only backs up the data area, when the app is deleted and restored from a backup it will lose the plugin and potentially also data.

[privsec]

@infogulch
could the backup locations be updated?

[murgero]

@robi Ah yes - this is what I was thinking about.

@mehdi - My apologies, I may have been thinking of alternate syntax to the symlinking command.

[MooCloud_Matt]

@girish

Probably a possible solution is to allow multiple Volumes to be mounted to the container, so there is no need for a syslink, and is still easy to backup, the backup script knows which volume to backup for each container.
Is a bit more work on the Backup script, but should allow an easier port to Cloudron.

Or, I see as an alternative the release of the Manifest standard as a separate Spec from Cloudron, like the "Compose Spec".
This should allow developers to better understand what cloudron and the manifest support and how they work, and hopefully start to act more in line with safer containerization standards.

[timconsidine]

@girish : it will for sure be impossible to persuade others to do it the Cloudron way, so the problem has to be addressed.

Personally I feel the Cloudron stability and reliability is SO IMPORTANT that it must be preserved and it is the priority. If it is impossible to find a solution, we have to accept a situation like "sorry, this XXX app is such a mess, it is not possible to make it comply with Cloudron standards, and even though it is good functionality, and we could possibly package it, we're not going to. Because it's a mess."

While I don't have a technical solution approach to the problem, I think it's almost inevitable that we end up with another status of apps :

• OFFICIAL (package tested, released, maintained, conforms to Cloudron standards)
• UNSTABLE (although I don't like this word : should be Beta or not-yet-officially-released)
• MESS or "Pig's Breakfast" (packaged, works, use it, but maintained as best endeavours, esp ref backups, because upstream dev does not conform to Cloudron standards)

Not ideal, but better than lowering Cloudron standards. Would rather accept not having XXX app in the store.

[mehdi]

I quite agree with @timconsidine . We could have these apps on Cloudron, but with a big fat red warning that says the app is unsupported :

• zero user support
• larger backups
• no auto-update

If unhappy: bother the original author.

[robi]

@moocloud_matt said in Deploying apps that treat code as data:

a possible solution is to allow multiple Volumes to be mounted to the container

This is along the lines of OpenEBS storage discussion here.

Elegant way to install an "elevator" to any directory in the container and swap out files we might need, without disrupting the app.

[MooCloud_Matt]

@robi
in general, having support for multiple drivers for Volumes is a good idea, but set in the Manifest standard the support for multiple volumes, could solve the issue at least partially.