Unsolved [BUG] Automount fail on reboot with Ubuntu 20.04
-
TO RESUME
- CONTEXT: While using an external storage (Over IP)
- ISSUE: DNS are not resolving since Cloudron use unbound in a container (which because operational after the initialization of the system)
- SOLUTION: adding in
/etc/hosts
the name and the IP
ORIGINAL POST
Basically I mount a SMB for my storage and after every reboot
I have to login via SSH and do amount -a
.note: I mount it directly in
/etc/fstab
because I have limited connection allowed. -
Do you see any failing attempt of automounting it during boot? Can you try adding
defaults,_netdev
to the fstab line? Maybe the network comes up a bit too slowly and thus the automount fails? -
@nebulon said in Why my CIFS automatic mount fail on every reboot ?:
Do you see any failing attempt of automounting it during boot?
I did manually look the /var/log/dmesg and saw nothing anormal
@nebulon said in Why my CIFS automatic mount fail on every reboot ?:
Can you try adding defaults,_netdev to the fstab line? Maybe the network comes up a bit too slowly and thus the automount fails?
I did add those line, without success
I still need to domount -a
after a rebootBTW it is on Ubuntu 20.04 LTS
-
Hm the exact same options work for me fine, however on 18.04 so this indeed might have changed in behavior, although I can't really find anything related to that.
-
@jodumont Me too I had this behavior on last restart...
-
It's working fine for me.
Here is what I have in fstab:
//uxxxxx.your-storagebox.de/backup /mnt/storage cifs iocharset=utf8,rw,credentials=/xxx/xxxxx-credentials.txt,uid=yellowtent,gid=yellowtent,file_mode=0660,dir_mode=0770 0 0
It did unmount once during a backup, but that's the only hiccup I've had to date.
-
@jdaviescoates are you with 18.04LTS or 20.04LTS ?
-
@jodumont 18.04. You?
-
@jdaviescoates said in Why my CIFS automatic mount fail on every reboot ?:
@jodumont 18.04. You?
jodumont said
BTW it is on Ubuntu 20.04 LTS
-
@jdaviescoates How is working now? Did you experienced problems in last days? I switched to Wasabi meanwhile find a durable solution to this problem...
-
@p44 I'm not having any problems. Streaming music off of my Hetzner Storage Box mounted as a Cloudron Volume and connected to my Navidrome app as I type.
-
If I am reading those version messages correctly, it does look like the automount on Ubuntu 20 has some kind of regression, while servers on 18 are working reliably?
-
@nebulon said in Why my CIFS automatic mount fail on every reboot ?:
If I am reading those version messages correctly, it does look like the automount on Ubuntu 20 has some kind of regression, while servers on 18 are working reliably?
In plain english
yes the issue seams to be only with Ubuntu 20.04
-
Possible related to this?
SMB1 disabled by default: can still be enabled via a /etc/samba/smb.conf config change;
https://wiki.ubuntu.com/FocalFossa/ReleaseNotes#Samba_4.11
More info here:
https://www.samba.org/samba/history/samba-4.11.0.htmlFound via https://askubuntu.com/questions/1229929/cant-acces-nas-anymore-after-upgrading-to-20-04
-
@jdaviescoates said in My CIFS automount fail on reboot with Ubuntu 20.04:
Possible related to this?
thank for your reach, I'll definitely give a try to this and come back soon.
-
@jdaviescoates I just look at it
and on Cloudron Samba server is not installed
so nothing to configure in /etc/samba/* which actually don't even exist.WEBDAV
I have the same behavior with webdav
on reboot I have to login and domount -a
here the option for my davfs in /etc/fstab:
rw,_netdev,uid=yellowtent,gid=yellowtent,file_mode=0660,dir_mode=0770 0 0
I also tried withauto
anddefaults
few people suggest to use rc.local
-
I just got an idea
@nebulon
it is possible than cloudron mount the volumes before my automount so then the target directory of my automount is not empty ?Also how could I destroy thoses volumes the GUI don't make it happen ?
-
@jodumont said in My CIFS automount fail on reboot with Ubuntu 20.04:
it is possible than cloudron mount the volumes before my automount so then the target directory of my automount is not empty ?
nope, even without any volumes configured in Cloudron
CIFS and WEBDAV won't mount automatically -
-
Cloudron itself does not handle any mount points as such. So this looks like this is some issue with 20.04 server then. Note that the desktop flavor has a lot more things installed usually, which may or may not trigger automounting correctly.
-
@nebulon said in [BUG] Automount fail on reboot with Ubuntu 20.04:
Cloudron itself does not handle any mount points as such. So this looks like this is some issue with 20.04 server then. Note that the desktop flavor has a lot more things installed usually, which may or may not trigger automounting correctly.
I have no doubt of the quality of coding from the Cloudron Team
but I just boot an ubuntu 20.04 LTS at Hetzner
didapt update apt install -y davfs2 mkdir /mnt/storagebox
than cut and past my 2 lines
1 from /etc/fstab
1 for /etc/davfs2/secretreboot and it work
so yes Ubuntu Desktop have probably more fuse than server but now the only difference is Cloudron.
-
thanks for testing, so then probably one of the dependencies we install, somehow changes either the init order or even disables some bits there. Looks like we have to debug this further then on fresh installations to get some more information what systemd does differently in both scenarios.
-
@nebulon said in [BUG] Automount fail on reboot with Ubuntu 20.04:
Looks like we have to debug this further then on fresh installations to get some more information what systemd does differently in both scenarios.
I'll run an fresh install of Cloudron than install davfs2 just to be sure
-
@jodumont said in [BUG] Automount fail on reboot with Ubuntu 20.04:
I'll run an fresh install of Cloudron than install davfs2 just to be sure
so boot up a new instance Ubuntu 20.04 LTS @Hetzner
ranwget https://cloudron.io/cloudron-setup chmod +x ./cloudron-setup ./cloudron-setup
reboot than install davfs2
apt install -y davfs2
and configure/etc/fstab
and/etc/davfs2/secrets
rebootit don't mount automatically
but mount without issue withmount -a
-
@jodumont I was able to reproduce this now also outside of hetzner on 20.04...not yet sure why and what causes the difference
-
To give some update, this is DNS related and how the init sequence works on 20.04 now.
Problem is at the point when systemd decides to attempt to mount the remote filesystems, unbound, the dns resolver is not yet started. This means the remote fs cannot be mounted.There are currently two workarounds:
- Instead of using the DNS name in the fstab entry, just use the IP
- add
x-systemd.automount
as an additional argument for the mountpoint in the fstab entry
Ideally we find a better flow by tweaking some of the init order in the future.
For now I've added that option requirement at https://docs.cloudron.io/backups/#cifs
-
@nebulon what about a retry after the initial failure after unbound loads?
maybe add a
mount -a
at the end of the unbound script? -
@nebulon said in [BUG] Automount fail on reboot with Ubuntu 20.04:
Instead of using the DNS name in the fstab entry, just use the IP
I'm old school, I prefer IP
So it is probably related to unbound-resolvconf no ?:Also on my side, by default my Hetzner NAS return me an IPv6, I didn't even know my Cloudron box as an IPv6
it is the same on your side ?
-
@robi said in [BUG] Automount fail on reboot with Ubuntu 20.04:
@nebulon what about a retry after the initial failure after unbound loads?
maybe add a mount -a at the end of the unbound script?How disabling the IPv6 in Cloudron
-
@jodumont said in [BUG] Automount fail on reboot with Ubuntu 20.04:
So it is probably related to unbound-resolvconf no ?:
Issue is related to Cloudron. We have an internal DNS server (unbound) and it's configured in such a way that it has to start after docker (very tricky to make it start before docker). Unfortunately, because the DNS starts only after docker, it's a bit too late for services like network mounts which start before them. Which is why changing the mount from name based to IP makes it all work.
I guess the fix is to change the way DNS server starts up but this is quite a complex task.
-
@nebulon said in [BUG] Automount fail on reboot with Ubuntu 20.04:
Instead of using the DNS name in the fstab entry, just use the IP
- with IPv6 I have this error:
/sbin/mount.davfs: invalid URL
- with IPv4 I have this error:
/sbin/mount.davfs: Mounting failed. 301 Moved Permanently
- with IPv6 I have this error:
-
@jodumont Oh.. maybe davfs requires the hostname because of vhost based configs! Can you try adding
x-systemd.automount
into the fstab entry instead? -
@nebulon said in [BUG] Automount fail on reboot with Ubuntu 20.04:
add x-systemd.automount as an additional argument for the mountpoint in the fstab entry
adding
x-systemd.automount,
in/etc/fstab
after_netdev,
work wellThanks for all of you (but specially @nebulon); without your help, I would probably be crying in a corner
-
@girish said in [BUG] Automount fail on reboot with Ubuntu 20.04:
@jodumont Oh.. maybe davfs requires the hostname because of vhost based configs! Can you try adding x-systemd.automount into the fstab entry instead?
it worked for one or two reboot, than I started adding Volumes in my Cloudron (not really sure if it related) and I noticed it stop.
- not on Cloudron, on my machine (but also Ubuntu 20.04LTS) it work well if I add user,noauto than mount it as a user.
- on proxmox (Debian 10) it work well with x-system-d.automount
Now CIFS or DAVFS2 with the option x-systemd.automount and even after I deleted all my Volumes in Cloudron WebUI have to same behavior (I need to login and do a mount -a)
-
The simpler solution would be to add the domains for mounts into the
/etc/hosts
file so no resolution is required. -
@robi said in [BUG] Automount fail on reboot with Ubuntu 20.04:
The simpler solution would be to add the domains for mounts into the /etc/hosts file so no resolution is required.
where were you ?
it is interesting more the technology become complex
more we forget about simple solution which was the norm 30 years agoFYI: I add both IPv6 and IPv4
-
@jodumont said in [BUG] Automount fail on reboot with Ubuntu 20.04:
where were you ?
I am here. Funny you should mention it.
Lol, I went to get some pizza today and got inspired.
Sometimes all we need is a break and some perspective.
Maybe that's a superpower.FYI: I add both IPv6 and IPv4
Well done. Curious which it actually uses to connect.
-
the problem with the storagebox might be an ip change, hetzner specifically states
It is very important to use the DNS name (.your-storagebox.de) instead of the IP address for your Storage Box; this is because the IP address can change. With the DNS address, you can access your Storage Box via IPv4 and IPv6.
-
@msbt v4 addresses changing I can understand, but v6 generally don't change much at all.
Also odd that a storage box would have dynamic addressing. Is there any additional info on why it's not static?
-
Using remote mounts also with volumes does add some possible race and inconsistency it looks like. So on Cloudron docker has to be started before unbound, now the remote mounting requires unbound for DNS resolving, however depending on how fast docker containers come up, they require volume mounts, which in turn, if depending on the remote mount points, we end up with some circular dependency.
The systemd automount would attempt to mount a remote once anything accesses the mountpoint. So if docker is quick enough and you have assigned some volumes from a mountpoint for a container, then again it would attempt to mount this before unbound is working.
For a start the automount will solve the issue of using a remote storage as a backup storage. When used with volumes, I think only using IPs will work for now.
-
@msbt said in [BUG] Automount fail on reboot with Ubuntu 20.04:
the problem with the storagebox might be an ip change, hetzner specifically states
they obviously say that simply to cover their ass in case of an power interruption, I don't really see when that would happen in other case.
@msbt I'm curious to read the whole chapter, where did you found this info ?
-
@robi said in [BUG] Automount fail on reboot with Ubuntu 20.04:
I am here. Funny you should mention it.
Lol, I went to get some pizza today and got inspired.I eat pizza last tonight, still not really inspired, but I installed Windows
-
@jodumont the word "hetzner" was a link to the docs where this is mentioned: https://docs.hetzner.com/robot/storage-box/general
-
@nebulon said in [BUG] Automount fail on reboot with Ubuntu 20.04:
Using remote mounts also with volumes does add some possible race and inconsistency it looks like
Not on the Cloudron side, but remotely NAS is in Germany, I'm in Bangkok I feel it and even for backup sometime the NAS is not available.
-
@jdaviescoates said in [BUG] Automount fail on reboot with Ubuntu 20.04:
Streaming music off of my Hetzner Storage Box mounted as a Cloudron Volume and connected to my Navidrome app as I type.
How it goes actually,
I'm trying to mount some directory from the NAS as volumes in CLOUDRON but most of the time apps (mainly tried with Nextcloud) start before so then it mess everything up
For now I mount it directly from Nextcloud interface but I can't share it with others apps.
My goal would be- saving money since 1TB is 10€ instead of 40€ if you build a volume of 1TB
- but also make apps interactive/sharing space such as: sync/uploading files with Nextcloud and make it available in Navidrome or Jellyfin
- I, aesthetically don't like to see the sharing arrow
@nebulon I wonder if it would be possible to transfer the mount volume principle inside the docker, at the container level ?
such as every docker will be the samba/davfs2 client and retain the info in their /etc/fstab ?? -
@jodumont I think you can raise this issue in Nextcloud forum to change the icon. TBH, I am not entirely sure why they are different icons. What does a normal user have to gain by knowing the distinction between a volume and a normal folder? Also, the arrow seems to suggest that it's going to pop up a new window or something, which I assume it doesn't.
-
@girish fair for the icons, and good point about open in a new tab
but what about the sharing volume in Cloudron ?
do you think would make sense to do the apps level or it will become too confusing and heavy on the system ? -
Those folder icons look like they are because they are shared folders.
-
@marcusquinn Oh really, then those arrows look even more confusing. Usually that icon (with the arrow) is associated with "sharing action" (i.e clicking it lets you share something) and not meant to indicate "shared items". But I am not a UI expert.
-
@girish and @nebulon, so do you think it is just me misusing the purpose of NAS / Volume in Cloudron or, I could have hope about this change ?
@jodumont said in [BUG] Automount fail on reboot with Ubuntu 20.04:
but what about the sharing volume in Cloudron ?
do you think would make sense to do the apps level or it will become too confusing and heavy on the system ? -
I will mark this as unsolved as per https://forum.cloudron.io/post/24684 . Currently, there is no solution to use DNS names in mounts, we have to rework our DNS system a bit to make it work.
-
@girish said in [BUG] Automount fail on reboot with Ubuntu 20.04:
Currently, there is no solution to use DNS names in mounts
don't worry, I understand that is an exception and probably not a priority, thanks for the consideration