Backup mount lost its permissions - unable to remount

[girish]

@jdaviescoates Is this a CIFS mount? What backup provider is this?

[avatar1024]

@jdaviescoates had a similar issue before and it was because the directory /mnt/cloudronbackup/ is supposed to be empty for volume to mount in it but it looks like you have a directory in it even if not mounted, no?

[jdaviescoates]

In the logs I spotted this:

Nov 29 10:26:43 box:shell remountMount spawn: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/remountmount.sh /mnt/cloudronbackup
Nov 29 10:26:43 box:shell remountMount (stdout): Job failed. See "journalctl -xe" for details.
Nov 29 10:26:43 box:shell remountMount code: 1, signal: null

So I looked at journalctl -xe:

root@v2202211129685206445:~# journalctl -xe
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- The unit UNIT has successfully entered the 'dead' state.
Nov 29 10:27:54 v2202211129685206445 systemd[1]: var-lib-docker-overlay2-127e343da78ef0ede10592cd097885b26c6f234a1b515015d0d7065b9272de0e-merged.mount: Succeeded.
-- Subject: Unit succeeded
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- The unit var-lib-docker-overlay2-127e343da78ef0ede10592cd097885b26c6f234a1b515015d0d7065b9272de0e-merged.mount has successfully entered the 'dead' state.
Nov 29 10:27:58 v2202211129685206445 kernel: Packet dropped: IN=eth0 OUT= MAC=26:49:b8:f9:a3:ae:2c:6b:f5:a0:77:c0:08:00 SRC=79.172.126.229 DST=89.58.52.168 LEN=52 TOS=0x00 PREC=0x00 TTL=117 ID=21873 PROTO=TCP S>
Nov 29 10:28:18 v2202211129685206445 containerd[723]: time="2022-11-29T10:28:18.029582762Z" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containe>
Nov 29 10:28:18 v2202211129685206445 containerd[723]: time="2022-11-29T10:28:18.029668956Z" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.contai>
Nov 29 10:28:18 v2202211129685206445 containerd[723]: time="2022-11-29T10:28:18.029685978Z" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.tt>
Nov 29 10:28:18 v2202211129685206445 containerd[723]: time="2022-11-29T10:28:18.029998514Z" level=info msg="starting signal loop" namespace=moby path=/run/containerd/io.containerd.runtime.v2.task/moby/a7d9d2180>
Nov 29 10:28:18 v2202211129685206445 systemd[31308]: run-docker-runtime\x2drunc-moby-a7d9d2180bced517a9f545671b092c95b00d42b4655039225c7312d78f562562-runc.Naz0a8.mount: Succeeded.
-- Subject: Unit succeeded
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- The unit UNIT has successfully entered the 'dead' state.
Nov 29 10:28:18 v2202211129685206445 systemd[1]: run-docker-runtime\x2drunc-moby-a7d9d2180bced517a9f545671b092c95b00d42b4655039225c7312d78f562562-runc.Naz0a8.mount: Succeeded.
-- Subject: Unit succeeded
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- The unit run-docker-runtime\x2drunc-moby-a7d9d2180bced517a9f545671b092c95b00d42b4655039225c7312d78f562562-runc.Naz0a8.mount has successfully entered the 'dead' state.
Nov 29 10:28:19 v2202211129685206445 containerd[723]: time="2022-11-29T10:28:19.142189066Z" level=info msg="shim disconnected" id=a7d9d2180bced517a9f545671b092c95b00d42b4655039225c7312d78f562562
Nov 29 10:28:19 v2202211129685206445 containerd[723]: time="2022-11-29T10:28:19.142750205Z" level=warning msg="cleaning up after shim disconnected" id=a7d9d2180bced517a9f545671b092c95b00d42b4655039225c7312d78f5>
Nov 29 10:28:19 v2202211129685206445 containerd[723]: time="2022-11-29T10:28:19.142840457Z" level=info msg="cleaning up dead shim"
Nov 29 10:28:19 v2202211129685206445 dockerd[980]: time="2022-11-29T10:28:19.142972057Z" level=info msg="ignoring event" container=a7d9d2180bced517a9f545671b092c95b00d42b4655039225c7312d78f562562 module=libcont>
Nov 29 10:28:19 v2202211129685206445 containerd[723]: time="2022-11-29T10:28:19.154428783Z" level=warning msg="cleanup warnings time=\"2022-11-29T10:28:19Z\" level=info msg=\"starting signal loop\" namespace=mo>
Nov 29 10:28:19 v2202211129685206445 systemd[31308]: var-lib-docker-overlay2-c549a5cfb9c43475404410d4b7449f8708c5e2e96188a9688dd10cf1a1b90ad2-merged.mount: Succeeded.
-- Subject: Unit succeeded
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- The unit UNIT has successfully entered the 'dead' state.
Nov 29 10:28:19 v2202211129685206445 systemd[1]: var-lib-docker-overlay2-c549a5cfb9c43475404410d4b7449f8708c5e2e96188a9688dd10cf1a1b90ad2-merged.mount: Succeeded.
-- Subject: Unit succeeded
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- The unit var-lib-docker-overlay2-c549a5cfb9c43475404410d4b7449f8708c5e2e96188a9688dd10cf1a1b90ad2-merged.mount has successfully entered the 'dead' state.
Nov 29 10:28:20 v2202211129685206445 containerd[723]: time="2022-11-29T10:28:20.157500189Z" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containe>
Nov 29 10:28:20 v2202211129685206445 containerd[723]: time="2022-11-29T10:28:20.157590962Z" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.contai>
Nov 29 10:28:20 v2202211129685206445 containerd[723]: time="2022-11-29T10:28:20.157605940Z" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.tt>
Nov 29 10:28:20 v2202211129685206445 containerd[723]: time="2022-11-29T10:28:20.161549653Z" level=info msg="starting signal loop" namespace=moby path=/run/containerd/io.containerd.runtime.v2.task/moby/5c2ae080b>
Nov 29 10:28:20 v2202211129685206445 systemd[1]: run-docker-runtime\x2drunc-moby-5c2ae080b598f606df18357e3f6acdc443e336296a5d1710583a2f7c58e1853e-runc.KEuTmD.mount: Succeeded.
-- Subject: Unit succeeded
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- The unit run-docker-runtime\x2drunc-moby-5c2ae080b598f606df18357e3f6acdc443e336296a5d1710583a2f7c58e1853e-runc.KEuTmD.mount has successfully entered the 'dead' state.

Given I've just done the kernel fix things in an attempt to fix the issues with Ubuntu 20.04 I wonder if it's something to do with:

Nov 29 10:27:58 v2202211129685206445 kernel: Packet dropped: IN=eth0 OUT= MAC=26:49:b8:f9:a3:ae:2c:6b:f5:a0:77:c0:08:00 SRC=79.172.126.229 DST=89.58.52.168 LEN=52 TOS=0x00 PREC=0x00 TTL=117 ID=21873 PROTO=TCP S>

?

Also, all this "dead" state docker stuff doesn't sound good?

[jdaviescoates]

@girish said in Backup mount lost its permissions:

@jdaviescoates Is this a CIFS mount? What backup provider is this?

Yes, Hetzner Storage Box.

[jdaviescoates]

@avatar1024 said in Backup mount lost its permissions:

@jdaviescoates had a similar issue before and it was because the directory /mnt/cloudronbackup/ is supposed to be empty for volume to mount in it but it looks like you have a directory in it even if not mounted, no?

It was empty the but the Cloudron error message told me to create it, so I did.

Now have no idea what to do

[jdaviescoates]

@jdaviescoates said in Backup mount lost its permissions - unable to remount:

Now have no idea what to do

I was about to rm -r the dir I created, but I'm scared if I do that then perhaps I'll end up deleting some backups... (because it now seems to have a /snapshop/ inside it although doesn't seem to actually be anything in there so perhaps I should just delete it and try again?)

[avatar1024]

@jdaviescoates humm ok. My storage box is mounted as sshfs so not sure how it translates to CIFS but in my case:

• if no volume is mounted /mnt/cloudronbackup is empty (and should be otherwise I also get a permission error when trying to mount)
• when mounted then /mnt/cloudronbackup and all sub directories become own by the hetzner username associated to the storage box

[avatar1024]

@jdaviescoates said in Backup mount lost its permissions - unable to remount:

I was about to rm -r the dir I created, but I'm scared if I do that then perhaps I'll end up deleting some backups...

Does the directory contains files? It shouldn't if your drive is not mounted!

[jdaviescoates]

@avatar1024 it has /snapshop/ in it, but looks like that is empty:

root@v2202211129685206445:/mnt/cloudronbackup/my.uniteddiversity.coop-netcup/snapshot# ls -la
total 8
drwxr-xr-x 2 yellowtent yellowtent 4096 Nov 29 10:24 .
drwxr-xr-x 3 yellowtent yellowtent 4096 Nov 29 10:25 ..

[avatar1024]

@jdaviescoates said in Backup mount lost its permissions - unable to remount:

it has /snapshop/ in it, but looks like that is empty:

Maybe wait for @staff to confirm but I'd say that you should deleted it all and try to remount your drive one more time. If that doesn't work (and it might not as you said it was empty originally) then I don't know...sorry.

[jdaviescoates]

@avatar1024 said in Backup mount lost its permissions - unable to remount:

(and it might not as you said it was empty originally)

At that point /cloudronbackup was root:root so perhaps that was the issue, but yeah I'll think I'll wait for @girish to chime in again.

Also thinking perhaps I should most to SSHFS too instead of CIFS given that seems to have resolved a bunch of issues for you, right?

[avatar1024]

@jdaviescoates said in Backup mount lost its permissions - unable to remount:

At that point /cloudronbackup was root:root so perhaps that was the issue

root:root is also the case for me, it is only the mount that changes the permissions to the storage box user.

@jdaviescoates said in Backup mount lost its permissions - unable to remount:

Also thinking perhaps I should most to SSHFS too instead of CIFS given that seems to have resolved a bunch of issues for you, right?

Yes sshfs has definitely be more reliable for me. I switched to it when I hit some unmounting / backup problems in the paste due to a bug in how CIFS were handle by Cloudron and never had a problem since so never went back to CIFS.

[jdaviescoates]

@avatar1024 thanks, I'll wait for @girish to chime in again and then likely look into mounting using SSHFS too...

[nebulon]

So for a start, check if this directory is actually mounted currently. You can do this with mountpoint /mnt/cloudronbackup/ and also systemctl status mnt-cloudronbackup.mount should give some status on that mointpoint. If it isn't mounted (it means it is just a local folder on the filesystem), then delete all contents of it (mostly empty folder I assume) and then try to remount via the button in your backs view of your Cloudron dashboard.

[jdaviescoates]

@nebulon said in Backup mount lost its permissions - unable to remount:

So for a start, check if this directory is actually mounted currently.

I don't really need to check that, because it isn't, that's the whole point.

But yeah, double triple confirmed:

root@v2202211129685206445:~# mountpoint /mnt/cloudronbackup/
/mnt/cloudronbackup/ is not a mountpoint
root@v2202211129685206445:~# systemctl status mnt-cloudronbackup.mount
● mnt-cloudronbackup.mount - backup
     Loaded: loaded (/etc/systemd/system/mnt-cloudronbackup.mount; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Tue 2022-11-29 10:26:43 UTC; 48min ago
      Where: /mnt/cloudronbackup
       What: //uXXXXXX.your-storagebox.de/backup

Nov 29 10:26:43 v2202211129685206445 systemd[1]: Mounting backup...
Nov 29 10:26:43 v2202211129685206445 mount[30438]: mount error(79): Can not access a needed shared library
Nov 29 10:26:43 v2202211129685206445 mount[30438]: Refer to the mount.cifs(8) manual page (e.g. man mount.cifs) and kernel log messages (dmesg)
Nov 29 10:26:43 v2202211129685206445 systemd[1]: mnt-cloudronbackup.mount: Mount process exited, code=exited, status=32/n/a
Nov 29 10:26:43 v2202211129685206445 systemd[1]: mnt-cloudronbackup.mount: Failed with result 'exit-code'.
Nov 29 10:26:43 v2202211129685206445 systemd[1]: Failed to mount backup.

@nebulon said in Backup mount lost its permissions - unable to remount:

If it isn't mounted (it means it is just a local folder on the filesystem), then delete all contents of it (mostly empty folder I assume) and then try to remount via the button in your backs view of your Cloudron dashboard.

Did that. Didn't work.

Then changed /mnt/cloudronbackup back to root:root (can you confirm what is should be?) to see if that would help. It didn't.

Also rebooted again for good measure, didn't help.

/mnt/cloudronbackup/ is not a mountpoint is still not a mountpoint.

[nebulon]

@jdaviescoates right so unless the mount succeeds that folder remains a local folder. It seems the underlying issue is mentioned there Can not access a needed shared library could it be that while you changed the kernel, apt had updated or autoremoved any extra libraries which cifs mount need? Also while clicking the remount button in the Cloudron dashboard, checkout the mentioned dmesg output, hopefully that reveals which shared library went missing on your system.

[jdaviescoates]

@nebulon said in Backup mount lost its permissions - unable to remount:

could it be that while you changed the kernel, apt had updated or autoremoved any extra libraries which cifs mount need?

I guess perhaps when I ran apt install linux-image-5.4.0-131-generic something like that could've happened?

Literally all I did was copy/ paste the instructions at https://forum.cloudron.io/post/57216

@nebulon said in Backup mount lost its permissions - unable to remount:

Also while clicking the remount button in the Cloudron dashboard, checkout the mentioned dmesg output, hopefully that reveals which shared library went missing on your system.

Spotted this:

[ 2001.662062] CIFS: Attempting to mount //uXXXXXX.your-storagebox.de/backup
[ 2001.662085] No dialect specified on mount. Default has changed to a more secure dialect, SMB2.1 or later (e.g. SMB3), from CIFS (SMB1). To use the less secure SMB1 dialect to access old servers which do not support SMB3 (or SMB2.1) specify vers=1.0 on mount.
[ 2001.663516] CIFS VFS: CIFS mount error: iocharset utf8 not found

?!?

I don't get how making the kernel changes would remove iocharset utf 8?

(also, please could you confirm what the permissions should be on all the folders within /mnt/ thanks!)

[girish]

@jdaviescoates Can you try apt install linux-modules-extra-5.4.0-131-generic and reboot ?

[jdaviescoates]

@girish said in Backup mount lost its permissions - unable to remount:

apt install linux-modules-extra-5.4.0-131-generic

Thanks that did the job!

Guess you need to add that to the fix instructions!

[girish]

@jdaviescoates that was quick!