Upgraded to 22.04 and 7.7.1 and cifs volumes are not mounting on boot

[ChristopherMag]

We just upgraded from Ubuntu 20.04 to 22.04 and from Cloudron 7.6.4 to 7.7.1.

We use the same cifs server for our backup destination and for two volumes. I have reentered all the information for them in their various screens after the upgrades just in case the configuration didn't come accross properly and everything works fine when it is first entered but if I reboot all of them have a red circle instead of the green one.

On the volumes screen if I hover over the red circle I see the following error message mount error(101): Network is unreachable Refer to the mount.cifs(8) manual page (e.g. man mount.cifs) and kernel log messages (dmesg)

Looking at journalctl -b | grep network | less I can see that the interface has link up but it takes 4 seconds later before it gets a dhcp address.

Looking at journalctl -b | grep mount | less I see that I get mount failures right after the link up but 4 seconds before the ip address is assigned via dhcp.

Clicking the remount volume button on the volumes page changes it from a red circle to a green one. Same happens on the backups page when clicking the remount storage button.

I see discussion here of people talking about using systemctl enable systemd-networkd-wait-online.service but I am not sure that would be directly relevant.

Is there something that can be changed so that the mount process waits until after an address has been assigned via dhcp?

[girish]

@ChristopherMag We use systemd to mount volumes. The error seems to indicate that the volumes are not mounting because maybe the network is not up yet when the mounting happens. I think your diagnosis to wait for network online is probably correct.

On Cloudron, the volumes need to get mounted before the docker containers. The container don't see the volume data if they get started before the volumes get mounted. For this reason, if we "block" in cifs volume service forever until the network comes up, the containers won't come up. The entire system them depends on whether the server comes online or not. We cannot add this limitation to Cloudron itself but in your situation maybe this is acceptable?

I found https://bbs.archlinux.org/viewtopic.php?id=284361 . Can you try if the suggested solution there would work for you ? (note that it is an arch forum and not ubuntu). Specifically ExecStartPre=/usr/bin/ping -qW 5 -c1 10.0.0.13 where the IP is CIFS Volume IP.

[ChristopherMag]

It seems like the systemd-networkd-wait-online.service service is supposed to ensure that the network is operational before proceeding with the rest of the startup process.

Looking at systemd-analyze blame | grep networkd-wait I see 49ms systemd-networkd-wait-online.service

Looking at the service unit file using nano /lib/systemd/system/systemd-networkd-wait-online.service I can see that this service runs the /lib/systemd/systemd-networkd-wait-online binary.

I am not sure the default behavior but looking around it seems that you can add specific options to define what interface /lib/systemd/systemd-networkd-wait-online will check and what state your looking for.

I have run sudo systemctl edit systemd-networkd-wait-online.service and then added:

[Service]
ExecStart=/lib/systemd/systemd-networkd-wait-online -i enp2s0:routable

This results in a new file /etc/systemd/system/systemd-networkd-wait-online.service.d/override.conf which contains the lines above in it.

After a reboot all the mountpoints are online without any action taken by me and if I run systemd-analyze blame | grep networkd-wait again I see 4.971s systemd-networkd-wait-online.service which seems to indicate that it is now waiting 4.9 seconds longer than it did before and presumably is now waiting until the interface is routable.

I am not sure why the default configuration of systemd-networkd-wait-online.service is not waiting for the dhcp client to finish receiving an ip address before it continues the boot process but I would hope there would be some configuration that could be made to say wait for at least one interface to be routable before proceeding or timeout after a specific period of time.

[girish]

@ChristopherMag good analysis! Where do we go from here? Do you think this is an ubuntu bug that it's not waiting for network online properly?

[ChristopherMag]

Doing some more digging it looks like you can define the state an adapter is required to be in for the systemd-networkd-wait-online.service to proceed by configuring the appropriate .network file.

On my system if I run cat /run/systemd/network/10-netplan-enp2s0.network I can see the following:

[Link]
RequiredForOnline=no

Per this suggestion I have run sudo nano /etc/systemd/network/20-enp2s0.network with the contents:

[Match]
Name=enp2s0

[Link]
RequiredForOnline=routable

I have run sudo systemctl revert systemd-networkd-wait-online.service to undo the override to systemd-networkd-wait-online done earlier and rebooted.

After reboot the mount points are not mounted and running cat /run/systemd/network/10-netplan-enp2s0.network I can see that the link is still set not to be required for online:

[Link]
RequiredForOnline=no

I think this is probably the right track and I am just missing something simple, will keep trying to find a way to override RequiredForOnline for a specific network link and then I think everything will work as expected.

[ChristopherMag]

What I have done that works for now is if I copy the output of cat /run/systemd/network/10-netplan-enp2s0.network into a file /etc/systemd/network/10-enp2s0.network and change the one line from RequiredForOnline=no to RequiredForOnline=routable and restart all mount points are available without having to be remounted.

If the new file has a prefix of 20 instead of 10 it doesn't work and the file in /etc/systemd/network seems to have no effect.

If I have a file with the 10 prefix and only include the partial adapter configuration of just setting RequiredForOnline without the rest of the adapter configuraiton when the system boots it will hang for 2 minutes waiting for the network to come up and once booted I can see that all the other network configuration is lost.

I thought the .network files are supposed to layer configuration on top of each other unless the file names exactly match.

I tried to find out where the settings from /run/systemd/network/10-netplan-enp2s0.network are coming from and have not been able to identify their source.

We can close this issue and I will pursue this further via serverfault or another context that more generally focuses on Ubuntu server administration.

Thanks for your help.

I feel like I must be missing something about how systemd networkd works.

[ChristopherMag]

For anyone coming later, Ubuntu server use's netplan for configuration via files in /etc/netplan and you should not configure networking directly via systemd-networkd .network files.

More details here.

In my case if I ran sudo cat /etc/netplan/00-installer-config.yaml I got the following:

# This is the network config written by 'subiquity'
network:
  ethernets:
    enp2s0:
      dhcp4: true
      optional: true
  version: 2

The specific issue was that I had optional: true which I had done when I first setup this system as I originally had it setup to use wireless while doing the initial configuration and then switched to hard wired once I deployed it.

Changing optional: true to optional: false, then removing the .network file I had put in /etc/systemd/network, and then rebooting I can now see that cat /run/systemd/network/10-netplan-enp2s0.network no longer has RequiredForOnline=no.

Apparently the default is to require the network adapter to be in a routable state so now with the correct netplan configuration I get the right generated systemd-networkd configuration which properly waits for the interface to be routable before trying to mount volumes and all volumes mount as expected without intervention.

The root cause behind this issue was me making a configuration change I meant to be temporary but then forgot to reverse and then because I didn't understand netplan my troubleshooting was in the wrong places.