[BUG] Automount fail on reboot with Ubuntu 20.04

JOduMonT

TO RESUME

CONTEXT: While using an external storage (Over IP)
ISSUE: DNS are not resolving since Cloudron use unbound in a container (which because operational after the initialization of the system)
SOLUTION: adding in /etc/hosts the name and the IP

ORIGINAL POST

Basically I mount a SMB for my storage and after every reboot
I have to login via SSH and do a mount -a.

note: I mount it directly in /etc/fstab because I have limited connection allowed.

nebulon

Do you see any failing attempt of automounting it during boot? Can you try adding defaults,_netdev to the fstab line? Maybe the network comes up a bit too slowly and thus the automount fails?

JOduMonT

@nebulon said in Why my CIFS automatic mount fail on every reboot ?:

Do you see any failing attempt of automounting it during boot?

I did manually look the /var/log/dmesg and saw nothing anormal

@nebulon said in Why my CIFS automatic mount fail on every reboot ?:

Can you try adding defaults,_netdev to the fstab line? Maybe the network comes up a bit too slowly and thus the automount fails?

I did add those line, without success
I still need to do mount -a after a reboot

BTW it is on Ubuntu 20.04 LTS

nebulon

Hm the exact same options work for me fine, however on 18.04 so this indeed might have changed in behavior, although I can't really find anything related to that.

p44

@jodumont Me too I had this behavior on last restart...

jdaviescoates

It's working fine for me.

Here is what I have in fstab:

//uxxxxx.your-storagebox.de/backup /mnt/storage cifs iocharset=utf8,rw,credentials=/xxx/xxxxx-credentials.txt,uid=yellowtent,gid=yellowtent,file_mode=0660,dir_mode=0770 0 0

It did unmount once during a backup, but that's the only hiccup I've had to date.

JOduMonT

@jdaviescoates are you with 18.04LTS or 20.04LTS ?

jdaviescoates

@jodumont 18.04. You?

JOduMonT

@jdaviescoates said in Why my CIFS automatic mount fail on every reboot ?:

@jodumont 18.04. You?

jodumont said

BTW it is on Ubuntu 20.04 LTS

p44

@jdaviescoates How is working now? Did you experienced problems in last days? I switched to Wasabi meanwhile find a durable solution to this problem...

jdaviescoates

@p44 I'm not having any problems. Streaming music off of my Hetzner Storage Box mounted as a Cloudron Volume and connected to my Navidrome app as I type.

nebulon

If I am reading those version messages correctly, it does look like the automount on Ubuntu 20 has some kind of regression, while servers on 18 are working reliably?

JOduMonT

@nebulon said in Why my CIFS automatic mount fail on every reboot ?:

If I am reading those version messages correctly, it does look like the automount on Ubuntu 20 has some kind of regression, while servers on 18 are working reliably?

In plain english yes the issue seams to be only with Ubuntu 20.04

jdaviescoates

Possible related to this?

SMB1 disabled by default: can still be enabled via a /etc/samba/smb.conf config change;

https://wiki.ubuntu.com/FocalFossa/ReleaseNotes#Samba_4.11

More info here:
https://www.samba.org/samba/history/samba-4.11.0.html

Found via https://askubuntu.com/questions/1229929/cant-acces-nas-anymore-after-upgrading-to-20-04

JOduMonT

@jdaviescoates said in My CIFS automount fail on reboot with Ubuntu 20.04:

Possible related to this?

thank for your reach, I'll definitely give a try to this and come back soon.

JOduMonT

@jdaviescoates I just look at it
and on Cloudron Samba server is not installed
so nothing to configure in /etc/samba/* which actually don't even exist.

WEBDAV

I have the same behavior with webdav
on reboot I have to login and do mount -a

here the option for my davfs in /etc/fstab: rw,_netdev,uid=yellowtent,gid=yellowtent,file_mode=0660,dir_mode=0770 0 0
I also tried with auto and defaults

few people suggest to use rc.local

JOduMonT

I just got an idea
@nebulon
it is possible than cloudron mount the volumes before my automount so then the target directory of my automount is not empty ?

Also how could I destroy thoses volumes the GUI don't make it happen ?

JOduMonT

@jodumont said in My CIFS automount fail on reboot with Ubuntu 20.04:

it is possible than cloudron mount the volumes before my automount so then the target directory of my automount is not empty ?

nope, even without any volumes configured in Cloudron
CIFS and WEBDAV won't mount automatically

JOduMonT

@nebulon and @girish

I just copy the config (/etc/fstab and /etc/davfs2/secret) to my Ubuntu 20.04LTS Desktop
and my webdav mount automatically on the reboot.

So it seams to be a bug on the cloudron side.

nebulon

Cloudron itself does not handle any mount points as such. So this looks like this is some issue with 20.04 server then. Note that the desktop flavor has a lot more things installed usually, which may or may not trigger automounting correctly.

JOduMonT

@nebulon said in [BUG] Automount fail on reboot with Ubuntu 20.04:

Cloudron itself does not handle any mount points as such. So this looks like this is some issue with 20.04 server then. Note that the desktop flavor has a lot more things installed usually, which may or may not trigger automounting correctly.

I have no doubt of the quality of coding from the Cloudron Team
but I just boot an ubuntu 20.04 LTS at Hetzner
did

apt update
apt install -y davfs2
mkdir /mnt/storagebox

than cut and past my 2 lines
1 from /etc/fstab
1 for /etc/davfs2/secret

reboot and it work

so yes Ubuntu Desktop have probably more fuse than server but now the only difference is Cloudron.

nebulon

thanks for testing, so then probably one of the dependencies we install, somehow changes either the init order or even disables some bits there. Looks like we have to debug this further then on fresh installations to get some more information what systemd does differently in both scenarios.

JOduMonT

@nebulon said in [BUG] Automount fail on reboot with Ubuntu 20.04:

Looks like we have to debug this further then on fresh installations to get some more information what systemd does differently in both scenarios.

I'll run an fresh install of Cloudron than install davfs2 just to be sure

JOduMonT

@jodumont said in [BUG] Automount fail on reboot with Ubuntu 20.04:

I'll run an fresh install of Cloudron than install davfs2 just to be sure

so boot up a new instance Ubuntu 20.04 LTS @Hetzner
ran

wget https://cloudron.io/cloudron-setup
chmod +x ./cloudron-setup
./cloudron-setup

reboot than install davfs2 apt install -y davfs2
and configure /etc/fstab and /etc/davfs2/secrets
reboot

it don't mount automatically
but mount without issue with mount -a

nebulon

@jodumont I was able to reproduce this now also outside of hetzner on 20.04...not yet sure why and what causes the difference

nebulon

To give some update, this is DNS related and how the init sequence works on 20.04 now.
Problem is at the point when systemd decides to attempt to mount the remote filesystems, unbound, the dns resolver is not yet started. This means the remote fs cannot be mounted.

There are currently two workarounds:

Instead of using the DNS name in the fstab entry, just use the IP
add x-systemd.automount as an additional argument for the mountpoint in the fstab entry

Ideally we find a better flow by tweaking some of the init order in the future.

For now I've added that option requirement at https://docs.cloudron.io/backups/#cifs

robi

@nebulon what about a retry after the initial failure after unbound loads?

maybe add a mount -a at the end of the unbound script?

JOduMonT

@nebulon said in [BUG] Automount fail on reboot with Ubuntu 20.04:

Instead of using the DNS name in the fstab entry, just use the IP

I'm old school, I prefer IP
So it is probably related to unbound-resolvconf no ?:

Also on my side, by default my Hetzner NAS return me an IPv6, I didn't even know my Cloudron box as an IPv6

it is the same on your side ?

JOduMonT

@robi said in [BUG] Automount fail on reboot with Ubuntu 20.04:

@nebulon what about a retry after the initial failure after unbound loads?
maybe add a mount -a at the end of the unbound script?

How disabling the IPv6 in Cloudron

girish

@jodumont said in [BUG] Automount fail on reboot with Ubuntu 20.04:

So it is probably related to unbound-resolvconf no ?:

Issue is related to Cloudron. We have an internal DNS server (unbound) and it's configured in such a way that it has to start after docker (very tricky to make it start before docker). Unfortunately, because the DNS starts only after docker, it's a bit too late for services like network mounts which start before them. Which is why changing the mount from name based to IP makes it all work.

I guess the fix is to change the way DNS server starts up but this is quite a complex task.

JOduMonT

@nebulon said in [BUG] Automount fail on reboot with Ubuntu 20.04:

Instead of using the DNS name in the fstab entry, just use the IP

with IPv6 I have this error: /sbin/mount.davfs: invalid URL
with IPv4 I have this error: /sbin/mount.davfs: Mounting failed. 301 Moved Permanently

girish

@jodumont Oh.. maybe davfs requires the hostname because of vhost based configs! Can you try adding x-systemd.automount into the fstab entry instead?

JOduMonT

@nebulon said in [BUG] Automount fail on reboot with Ubuntu 20.04:

add x-systemd.automount as an additional argument for the mountpoint in the fstab entry

adding x-systemd.automount, in /etc/fstab after _netdev, work well

Thanks for all of you (but specially @nebulon); without your help, I would probably be crying in a corner

JOduMonT

@girish said in [BUG] Automount fail on reboot with Ubuntu 20.04:

@jodumont Oh.. maybe davfs requires the hostname because of vhost based configs! Can you try adding x-systemd.automount into the fstab entry instead?

it worked for one or two reboot, than I started adding Volumes in my Cloudron (not really sure if it related) and I noticed it stop.

not on Cloudron, on my machine (but also Ubuntu 20.04LTS) it work well if I add user,noauto than mount it as a user.
on proxmox (Debian 10) it work well with x-system-d.automount

Now CIFS or DAVFS2 with the option x-systemd.automount and even after I deleted all my Volumes in Cloudron WebUI have to same behavior (I need to login and do a mount -a)

robi

The simpler solution would be to add the domains for mounts into the /etc/hosts file so no resolution is required.

JOduMonT

@robi said in [BUG] Automount fail on reboot with Ubuntu 20.04:

The simpler solution would be to add the domains for mounts into the /etc/hosts file so no resolution is required.

where were you ?
it is interesting more the technology become complex
more we forget about simple solution which was the norm 30 years ago

FYI: I add both IPv6 and IPv4

robi

@jodumont said in [BUG] Automount fail on reboot with Ubuntu 20.04:

where were you ?

I am here. Funny you should mention it.
Lol, I went to get some pizza today and got inspired.
Sometimes all we need is a break and some perspective.
Maybe that's a superpower.

FYI: I add both IPv6 and IPv4

Well done. Curious which it actually uses to connect.

msbt

the problem with the storagebox might be an ip change, hetzner specifically states

It is very important to use the DNS name (.your-storagebox.de) instead of the IP address for your Storage Box; this is because the IP address can change. With the DNS address, you can access your Storage Box via IPv4 and IPv6.

robi

@msbt v4 addresses changing I can understand, but v6 generally don't change much at all.

Also odd that a storage box would have dynamic addressing. Is there any additional info on why it's not static?

nebulon

Using remote mounts also with volumes does add some possible race and inconsistency it looks like. So on Cloudron docker has to be started before unbound, now the remote mounting requires unbound for DNS resolving, however depending on how fast docker containers come up, they require volume mounts, which in turn, if depending on the remote mount points, we end up with some circular dependency.

The systemd automount would attempt to mount a remote once anything accesses the mountpoint. So if docker is quick enough and you have assigned some volumes from a mountpoint for a container, then again it would attempt to mount this before unbound is working.

For a start the automount will solve the issue of using a remote storage as a backup storage. When used with volumes, I think only using IPs will work for now.

JOduMonT

@msbt said in [BUG] Automount fail on reboot with Ubuntu 20.04:

the problem with the storagebox might be an ip change, hetzner specifically states

they obviously say that simply to cover their ass in case of an power interruption, I don't really see when that would happen in other case.

@msbt I'm curious to read the whole chapter, where did you found this info ?

JOduMonT

@robi said in [BUG] Automount fail on reboot with Ubuntu 20.04:

I am here. Funny you should mention it.
Lol, I went to get some pizza today and got inspired.

I eat pizza last tonight, still not really inspired, but I installed Windows

msbt

@jodumont the word "hetzner" was a link to the docs where this is mentioned: https://docs.hetzner.com/robot/storage-box/general

JOduMonT

@nebulon said in [BUG] Automount fail on reboot with Ubuntu 20.04:

Using remote mounts also with volumes does add some possible race and inconsistency it looks like

Not on the Cloudron side, but remotely NAS is in Germany, I'm in Bangkok I feel it and even for backup sometime the NAS is not available.

JOduMonT

@jdaviescoates said in [BUG] Automount fail on reboot with Ubuntu 20.04:

Streaming music off of my Hetzner Storage Box mounted as a Cloudron Volume and connected to my Navidrome app as I type.

How it goes actually,
I'm trying to mount some directory from the NAS as volumes in CLOUDRON but most of the time apps (mainly tried with Nextcloud) start before so then it mess everything up
For now I mount it directly from Nextcloud interface but I can't share it with others apps.
My goal would be

saving money since 1TB is 10€ instead of 40€ if you build a volume of 1TB
but also make apps interactive/sharing space such as: sync/uploading files with Nextcloud and make it available in Navidrome or Jellyfin
I, aesthetically don't like to see the sharing arrow

@nebulon I wonder if it would be possible to transfer the mount volume principle inside the docker, at the container level ?
such as every docker will be the samba/davfs2 client and retain the info in their /etc/fstab ??

girish

@jodumont I think you can raise this issue in Nextcloud forum to change the icon. TBH, I am not entirely sure why they are different icons. What does a normal user have to gain by knowing the distinction between a volume and a normal folder? Also, the arrow seems to suggest that it's going to pop up a new window or something, which I assume it doesn't.

JOduMonT

@girish fair for the icons, and good point about open in a new tab
but what about the sharing volume in Cloudron ?
do you think would make sense to do the apps level or it will become too confusing and heavy on the system ?

marcusquinn

Those folder icons look like they are because they are shared folders.

girish

@marcusquinn Oh really, then those arrows look even more confusing. Usually that icon (with the arrow) is associated with "sharing action" (i.e clicking it lets you share something) and not meant to indicate "shared items". But I am not a UI expert.

JOduMonT

@girish and @nebulon, so do you think it is just me misusing the purpose of NAS / Volume in Cloudron or, I could have hope about this change ?

@jodumont said in [BUG] Automount fail on reboot with Ubuntu 20.04:

but what about the sharing volume in Cloudron ?
do you think would make sense to do the apps level or it will become too confusing and heavy on the system ?

girish

I will mark this as unsolved as per https://forum.cloudron.io/post/24684 . Currently, there is no solution to use DNS names in mounts, we have to rework our DNS system a bit to make it work.

JOduMonT

@girish said in [BUG] Automount fail on reboot with Ubuntu 20.04:

Currently, there is no solution to use DNS names in mounts

don't worry, I understand that is an exception and probably not a priority, thanks for the consideration

Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.

Unsolved [BUG] Automount fail on reboot with Ubuntu 20.04

TO RESUME

ORIGINAL POST

WEBDAV

Now CIFS or DAVFS2 with the option x-systemd.automount and even after I deleted all my Volumes in Cloudron WebUI have to same behavior (I need to login and do a mount -a)