"Failed to open stream: Permission denied" for cache/data

[neoplex]

Anyone else seeing this behavior?

Having trouble with files in the cache directory under /app/data/storage/framework/cache/data being created with root/root, stalling background jobs and causing frequent disconnects in the frontend.

After changing the user/group for cache/data directory recursively to www-data/www-data, things are working for a moment. Then the same thing happens again. I had a quick go around and couldn't find any obvious misalignment in the configuration or running processes.

[joseph]

AFAICT, everything in the container runs as www-data user. Not sure who is creating these. Can you make a note of the timestamps of the bad directories, next time around? We should try to correlate with app logs.

[neoplex]

It appears that the directories owned by root are created either on the hour or at 45 minutes past the hour. And it seems to be related to those stalled App\Jobs\UpdateFolderCounters job.

[joseph]

@neoplex are you able to reproduce this by running the cron task manually? You can do this by Web Terminal -> Cron on the top. thanks

[neoplex]

It's not the cron job. It's this job that's executed by the queue worker.

Although the PHP script that the queue worker is being executed with runs as www-data.

The timestamps match.

[neoplex]

[joseph]

are you on the package 1.13.2 ? just to get that out of the way.

[neoplex]

Yep!

[joseph]

@neoplex unfortunately, cannot reproduce. If it's easy to see this, can you please send a mail to support@cloudron.io and we can take a look to debug this further.

[neoplex]

I've fixed this. Will send a patch via existing support email thread.

[neoplex]

I am also going to leave this here:

I finally got around to putting together a fix:

https://github.com/pronetivity/cloudron-freescout/commit/27697c310fb373f1a972c7990f5c00dc3052ee54

A detailed analysis is available here:

https://github.com/pronetivity/cloudron-freescout/blob/master/LARAVEL-CACHE-FIX.md

Disclaimer: the commit also includes a few quality-of-life changes. The actual fix is limited to the caching adjustments.

Cheers,
JD

[girish]

@neoplex thank you. Since we host on gitlab, I applied the patch by hand with only the relevant bits - https://git.cloudron.io/packages/freescout-app/-/merge_requests/61

[neoplex]

Hey @girish, thanks for picking up the patch!

So, that one cleaned up the queue worker side of things (which was genuinely broken), but the root-owned cache files started appearing again. I've spent some more time on this and I FINALLY found the culprit ...

Turns out it's not anything inside the app container. It's the scheduler sidecar.

On the host:

$ docker ps --format '{{.Names}} {{.Command}}' | grep <app-id>
<app-id>-crontab.0  "/bin/sh -c 'php /ap…"
<app-id>             "/app/pkg/start.sh"

$ docker inspect --format '{{.Config.Cmd}}' <app-id>-crontab.0
[/bin/sh -c php /app/code/artisan schedule:run >> /dev/null 2>&1]

$ docker inspect --format '{{.Config.User}}' <app-id>-crontab.0
(empty - runs as root)

The sidecar runs php artisan schedule:run directly as root every minute, creating scheduler mutex files and other cache entries under storage/framework/cache/data/ owned by root:root. When the app (running as www-data) tries to write to those same directories - permission denied.

Two things I noticed:

1. The sidecar doesn't use the manifest's "command": "/app/pkg/cron.sh" - which uses gosu to drop privileges - it hardcodes php artisan schedule:run instead
2. It runs without a user set, so it defaults to root

The fix from MR61 is still good to keep, but need to address the sidecar situation. For the freescout app specifically, I've now removed the scheduler addon and switched to running the schedule:run job in the same container via a supervisor-managed process. That eliminates the sidecar container entirely.

I've patched and tested the repo accordingly:

https://github.com/pronetivity/cloudron-freescout/commit/6771b826ee45cca6fc75d145f85d9d3da198daae

https://github.com/pronetivity/cloudron-freescout/blob/master/LARAVEL-CACHE-FIX.md

As for the scheduler sidecar - shouldn't it respect the manifest command (or run as the app user)? That would be a nicer fix at the platform level going forward.

Cheers,
JD

[girish]

@neoplex not sure I understand. cron.sh does run as root. This is intentional because an app can run with multiple users. The cron.sh can decide what to do.

https://git.cloudron.io/packages/freescout-app/-/blob/master/cron.sh?ref_type=heads#L10 is running it via gosu which basically is run-as "www-data". The cron script is thus not being run as root.

[neoplex]

@girish you're right that cron.sh itself uses gosu. That part is fine. The issue isn't with cron.sh.

The issue is the sidecar container that the scheduler addon creates. On the host:

$ docker ps --format '{{.Names}} {{.Command}}' | grep 1af144bb
1af144bb-fbf4-434d-8edd-bb4b95c00ef5-crontab.0  "/bin/sh -c 'php /ap…"
1af144bb-fbf4-434d-8edd-bb4b95c00ef5             "/app/pkg/start.sh"

The sidecar doesn't run /app/pkg/cron.sh. It runs php artisan schedule:run directly:

$ docker inspect --format 'Cmd: {{.Config.Cmd}}
  User: "{{.Config.User}}"
  Image: {{.Config.Image}}' 1af144bb-fbf4-434d-8edd-bb4b95c00ef5-crontab.0

Cmd: [/bin/sh -c php /app/code/artisan schedule:run >> /dev/null 2>&1]
User: ""
Image: cloudron/net.freescout.cloudronapp:202603171110280000

No user is set (empty string), so it defaults to root. The gosu in cron.sh is never reached because the sidecar bypasses cron.sh entirely.

This is easy to verify on any box with a freescout app running:

docker inspect --format '{{.Config.Cmd}}' <app-id>-crontab.0

This is what creates the root-owned cache files under storage/framework/cache/data/.

[neoplex]

Wait one, I have a suspicion ...

[neoplex]

@girish apologies, I jumped the gun on that last post. You were right.

I had a look at the box source code (src/scheduler.js, src/docker.js) and the manifest command IS respected by the scheduler. The php artisan schedule:run on my instance came from a custom crontab entry carried over when we migrated from a standalone install to Cloudron over 2 years ago. That entry must have slipped through unnoticed, but it explains everything.

After removing it, the scheduler container (now suffixed -housekeeping instead of -crontab.0) correctly runs cron.sh:

$ docker inspect --format '{{.Config.Cmd}}' 1af144bb-fbf4-434d-8edd-bb4b95c00ef5-housekeeping
Cmd: [/bin/sh -c /app/pkg/cron.sh]

Things are working correctly with cron.sh + gosu now

I should have caught this sooner. In my defense (barely), the Cron tab of the app doesn't mention that commands run as root. I eventually found it in the documentation at the bottom of the Cron page. A small note in the Cron tab itself would probably help others avoid the same mistake.

Cheers,
JD