@girish Your question got me wondering what was going on.

Turns out, two apps on the volume both had caches that were insane (one due to an old uninstalled WP cache plugin that was removed, but maintained its 12 MILLION cached files, and the other was Mastodon, which needed some manual cleaning for the 800k files it had cached).

Turns out there were just SO MANY folders/files that any attempt to backup or read the size would just time out.

D'oh! 😖

@privsec you can play with rclone mounts for this, however you want to minimize the network distance to avoid hiccups.

Ah, I guess the error message could be more descriptive. Currently there is a hard-coded allowlist of paths on the filessystem to be used as volumes, to avoid accidents where someone might add system folders. The list is at https://git.cloudron.io/cloudron/box/-/blob/master/src/volumes.js#L57

@zion I am not sure I fully understand your process there, but generally a disk partition is just a section of a hard drive. That section then has to be formatted with a filesystem, in this case ext4. This can be done via mkfs.ext4 /dev/<you partition identifier>. Once that is done, you should be able to just use the Cloudron dashboard UI to add it then. This makes Cloudron create the mountpoint for you.

@nebulon Thank you. After checking, it turned out that the issue is about efs not mounted.

It seems to be messed up with security groups and/or network interfaces (eni). Hence closing this threat for now. Need to dig deeper and probably understand the concepts of security groups and eni better in AWS.

Thanks for your help!

Edit: Just realizing that can't set it to 'solved'

@girish Thanks for taking the time to ponder on this !

Yes you got me right and I understand why it's not possible in the case of nocodb.

just FYI, the path nocodb use in the cloudron context to store uploaded files is
/app/data/nc/uploads/noco/{NameOfTheDatabase}

f9e7d383-3c20-4693-9060-caf2008b655f-image.png

Your suggestion to link directly to the LAMP or to a static "mirror" of this is the path I finally went with, it's was the logical step since provided I can link to it, where the media is does not matter much.

i'm not yet releasing this publicly but it's coming !

Thanks for the help!

Maybe there was some bug in our migration path. Will keep an eye out for related bug reports.

@msbt We won't break it intentionally atleast.

@SamGreenwood please give us more context via screenshots of what you're looking at.

In general cloudron won't allow you to map volumes if they're not there, or move storage to the wrong mount points.

@girish This makes a ton of sense, although I'm not quite sure if I'm doing it right. Under what you linked to, the section should be under the Resources tab, although I think it's moved to the Storage tab (no problems with that, I just wanted to make sure I'm handling it correctly).

2dfb1d41-c351-4f45-8352-99a9505b43d2-image.png

99f01fc2-bf27-42bb-8de1-dc11eaff4e2b-image.png

The part that's a little confusing is how 'App Data' and Volumes tie together in storage (now it seems this may be resolved as you said, since your working on Volume updating at some point in the future).

Anyways, App Data says to mount an external EXT4 disk, which I assume is done with the 'Mounts' interface right below it. However, the mount says that you are mounting 'Volumes'.

The documentation says 'App Data' is backed up. However, the documentation also says that 'Volumes' are not backed up. So that does leave me a little perplexed as to whether the data will be backed up or not. I think according to your comment, that it will be backed up (because you did say it would no matter where it is on the server). But I just thought I'd check 🙂

In any case, thanks for your help, and hope you have an awesome rest of your week.

In 7.1 atleast, this seems fine:

1c2d66a7-2e5f-4590-b935-685d949310b8-image.png

@nebulon Had to wait a few days to execute on moved cloudron server.
all resolved now.
Thank you for this.

@fbartels The thing is I can't check since the volume now works for me. But what I'm saying is that the issue was wider than just not being to write with the NC app, since data in the volume was not read / written into correctly anywhere (in NC app, in NC terminal and in Cloudron file browser) and that has nothing to do with the www-data user as far as I can tell.

And by the way you are right that the user in the terminal is root so it's true that being able to write with the terminal could still mean there is a restriction on the folder which would prevent the NC app to write. But as explained above, I don't believe this was the problem in my case.

Ah, thanks for reporting. I have opened https://git.cloudron.io/cloudron/box/-/issues/808 internally .

@girish since it worked with my other NAS, I am pretty confident now that it is rather a Permission-Issue on the TrueNAS-Side which I dont understand well enough as of yet.

Thanks for the link!

@3246 @girish , I am not sure (didn't try myself) but things suggested in this post may help ..

@mehdi thanks for reporting, this is fixed now. Next time a separate thread might be better to track those things if possible.

Amazing, and makes all the difference!

@nebulon Yeah I went so far to migrate to a fresh ubuntu installation to find out if there is something wrong on a system level but that didn't help either. Moving to an external drive did work though for some reason. But that's not what I want since that is a much more expensive solution…

https://forum.cloudron.io/topic/7619/filesystem-mount-inside-container for others to follow along.

The full stack trace is:

16/07/2021 17:58:24 :: [console] Error writing to collectd.localhost.df-sdc1.df_complex-used: Unable to read header (/var/lib/graphite/whisper/collectd/localhost/df-sdc1/df_complex-used.wsp) 16/07/2021 17:58:24 :: [console] Unhandled Error Traceback (most recent call last): File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 266, in <lambda> inContext.theWork = lambda: context.call(ctx, func, *args, **kw) File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 122, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 85, in callWithContext return func(*args,**kw) File "/usr/lib/python3/dist-packages/carbon/writer.py", line 189, in writeForever writeCachedDataPoints() --- <exception caught here> --- File "/usr/lib/python3/dist-packages/carbon/writer.py", line 165, in writeCachedDataPoints state.database.write(metric, datapoints) File "/usr/lib/python3/dist-packages/carbon/database.py", line 124, in write whisper.update_many(path, datapoints) File "/usr/lib/python3/dist-packages/whisper.py", line 740, in update_many return file_update_many(fh, points, now) File "/usr/lib/python3/dist-packages/whisper.py", line 747, in file_update_many header = __readHeader(fh) File "/usr/lib/python3/dist-packages/whisper.py", line 294, in __readHeader raise CorruptWhisperFile("Unable to read header", fh.name) whisper.CorruptWhisperFile: Unable to read header (/var/lib/graphite/whisper/collectd/localhost/df-sdc1/df_complex-free.wsp)

The last line gives a hint that the graphite file is corrupt. So, I removed all the whisper files in /home/yellowtent/platformdata/graphite/whisper/collectd/localhost/df-sdc1 and graphs seems to work after that.

@atridad Maybe a good idea to check the health of the hard disk ? Also, since we hit the systemd issue on the same server.

@girish HAH nevermind. Its me being a Linux noob...

turns out I mounted the drive itself not the partition. The files are there.

@jdaviescoates said in Volume disk space strangeness:

/me goes to reduce the size of the volume back to a perfectly sufficient 10 GB...

Oh, bollocks. You can't decrease the size of Hetzner Cloud Volumes, only increase them. So I'd have to create a whole new one and then and then and then...can't be bothered with all that so I guess I'll just stomach the slightly increased cost of 15 GB over 10 GB.

@odie Still stuck on this. The culprit is definitely that the usb network card fails to receive its configuration on boot. I cannot get netmanager to configure and initialize it at all. Only the two manual commands seem to work:

sudo ip addr add 192.168.9.101/24 dev enxc4411eb4c476 sudo ip link set dev enxc4411eb4c476 up

I have tried various thing with network manager, and I've tried adding a config file to systemd-networkd under /etc/systemd/network/ - the only thing I achieved, was to have every network freeze when I inserted the usb ethernet adapter (only to unfreeze as soon as I disconnected it). Tried keeping it disconnected for longer, just to see, but connections were frozen until I unplugged the usb network card. So I had to remove these config files.

Can anyone offer suggestions? I don't know where to even look for assistance on this now... Thanks!

@drpaneas Ah thanks, I think I found the bug. It seems a "permission denied" issue is being flagged incorrectly as directory does not exist. The chown you did essentially changed permissions and fixed the issue.

@annaooo To add to this, it's best to create shared folders under /srv, /mnt, /opt instead of /home/yellowtent. The /home/yellowtent is the home directory of the user which cloudron platform runs as and it's not meant for user data. In fact, it won't let you create a volume under /home/yellowtent.

@girish said in See volume disk space usage on System Info:

This is fixed in 6.1 - https://git.cloudron.io/cloudron/box/-/issues/756

I see one of my volumes (an ext4 Hetzner Cloud Volume), but not the other (CIFS Hetzner Storage Box).

Is there any way to display disk usage of CIFS mounts too?

Thanks!

@mario said in it is possible to share data between apps ?:

Through volumes, sure.

yes, I just found it, this is why it was familiar to me because it was mentioned in the Cloudron 6 release note 😉

@nebulon for me it is no issue anymore because of the Volumes. In this Minio app I don't share the Volume with other apps and it's not backed up just like I want.

Hopefully in future releases, like @girish wrote, when they will be backed up there will also be an option to never backup specific Volumes?

@girish it works like a charm, absolutely awsome!