NFS mount for Apps
-
I have a few NFS mounts on my server for videos, music, etc., which I pass on to apps (Jellyfin, Navidrome, etc.). I am curious as to how these mounts are passed to apps as they start to run. When I reboot Cloudron, these apps run but none of the mounts are available. I mean I can't see the videos or music shares. I have to reboot the app to get access to these folders. So, are the apps starting before the NFS shares get mounted? If so, is there a way to delay app start until the shares are available?
-
Are you using the volume feature of Cloudron for the NFS mounts and have you connected those volumes to that app?
@nebulon yes, I am using the volumes feature. I then, simply pass these mounts to apps in the Storage area as read only.
-
I have a few NFS mounts on my server for videos, music, etc., which I pass on to apps (Jellyfin, Navidrome, etc.). I am curious as to how these mounts are passed to apps as they start to run. When I reboot Cloudron, these apps run but none of the mounts are available. I mean I can't see the videos or music shares. I have to reboot the app to get access to these folders. So, are the apps starting before the NFS shares get mounted? If so, is there a way to delay app start until the shares are available?
@neurokrish next time you hit this, can you also check if the files are not visible in the volume's file manager as well (there is a file manager button next to the right of the volume in the Volumes UI) ? If it's visible, it indicates that there is a race between the mounting and the container getting created.
As an idea, the systemd mount points are created under
/etc/systemd/system/. Can you try editing the unit files there to haveBefore=docker.serviceand see if that helps?(Maybe we need to create "wait for" unit files like this - https://serverfault.com/questions/904421/docker-service-starts-before-zfs )
-
I have a similar issue with my sshfs mount for my Nextcloud instances (setup described here). On every reboot, I have to manually remove both subfolders from the where the mount should be (empty directories) and run mount -a manually to get the correct content online.
So the sshfs mount is at
/mnt/cloudand there are 2 volumes set up like this:
After reboot, there are those two empty mount points in there that block the actual mounting (
/mnt/cloud/cloud1and/mnt/cloud/cloud2). Those two I have to rm -r in order to get the mounts back. Hope that makes sense. -
@neurokrish next time you hit this, can you also check if the files are not visible in the volume's file manager as well (there is a file manager button next to the right of the volume in the Volumes UI) ? If it's visible, it indicates that there is a race between the mounting and the container getting created.
As an idea, the systemd mount points are created under
/etc/systemd/system/. Can you try editing the unit files there to haveBefore=docker.serviceand see if that helps?(Maybe we need to create "wait for" unit files like this - https://serverfault.com/questions/904421/docker-service-starts-before-zfs )
@girish sorry, as I mentioned in my other post "Can't access Cloudron", it appears that my ISP started blocking ports 80 and 443. So, I can't troubleshoot the NFS mount issue. I am, however, pretty sure that this problem exists!
Let me know if I can do anything to troubleshoot without access to my Cloudron dashboard, and I will.
-
@neurokrish next time you hit this, can you also check if the files are not visible in the volume's file manager as well (there is a file manager button next to the right of the volume in the Volumes UI) ? If it's visible, it indicates that there is a race between the mounting and the container getting created.
As an idea, the systemd mount points are created under
/etc/systemd/system/. Can you try editing the unit files there to haveBefore=docker.serviceand see if that helps?(Maybe we need to create "wait for" unit files like this - https://serverfault.com/questions/904421/docker-service-starts-before-zfs )
@girish I am back online with Cloudron. Looks like I am caught in a bit of a crossfire with my ISP meddling with things. I checked in the file manager near the mounts in Volumes UI. There are no files over there. Have a look,
I will edit the files you suggested and report back!
-
@neurokrish next time you hit this, can you also check if the files are not visible in the volume's file manager as well (there is a file manager button next to the right of the volume in the Volumes UI) ? If it's visible, it indicates that there is a race between the mounting and the container getting created.
As an idea, the systemd mount points are created under
/etc/systemd/system/. Can you try editing the unit files there to haveBefore=docker.serviceand see if that helps?(Maybe we need to create "wait for" unit files like this - https://serverfault.com/questions/904421/docker-service-starts-before-zfs )
@girish editing the unit files to add
Before=docker.servicedid not work for me. I can't see any of my files. However, if I try to mount these shares in the host system by editing fstab file, they are available on boot.I can also confirm @msbt 's observations. If after boot, I go to the Volumes UI and delete, re-add shares, all these files become available. I then have to go the app (Jellyfin, Navidrome, etc.) and reboot the app to see the shares..
-
@girish editing the unit files to add
Before=docker.servicedid not work for me. I can't see any of my files. However, if I try to mount these shares in the host system by editing fstab file, they are available on boot.I can also confirm @msbt 's observations. If after boot, I go to the Volumes UI and delete, re-add shares, all these files become available. I then have to go the app (Jellyfin, Navidrome, etc.) and reboot the app to see the shares..
I am just putting this information out here for anybody else. To solve this issue, I created a systemd drop-in unit for docker service to wait for NFS mounts to come online as mentioned in this post https://thedaneshproject.com/posts/start-docker-after-nfs-mounts-come-online/ .
I did not yet test on Volumes in Cloudron but in my case the NFS folders are mounted in the host under
/mediaand these folders are added as mounts in the Volume section of Cloudron (as Filesystem Mountpoint).Perhaps @girish @nebulon can implement something similar in Cloudron at some point!
-
I am just putting this information out here for anybody else. To solve this issue, I created a systemd drop-in unit for docker service to wait for NFS mounts to come online as mentioned in this post https://thedaneshproject.com/posts/start-docker-after-nfs-mounts-come-online/ .
I did not yet test on Volumes in Cloudron but in my case the NFS folders are mounted in the host under
/mediaand these folders are added as mounts in the Volume section of Cloudron (as Filesystem Mountpoint).Perhaps @girish @nebulon can implement something similar in Cloudron at some point!
@neurokrish That's great, can you post the contents of that file?
Then others can take it further from there.
-
@neurokrish That's great, can you post the contents of that file?
Then others can take it further from there.
@robi sure, I created a file, let's say,
x.confin/etc/systemd/system/docker.service.das mentioned in the link posted above. The contents of that file are,
media-movies.mount is just my systemd unit for the NFS mount. I have rebooted my server a few times after this change and my files are all accessible in apps as desired!
-
@robi sure, I created a file, let's say,
x.confin/etc/systemd/system/docker.service.das mentioned in the link posted above. The contents of that file are,media-movies.mount is just my systemd unit for the NFS mount. I have rebooted my server a few times after this change and my files are all accessible in apps as desired!
@neurokrish Sounds like it also needs the contents of what that references.
-
@neurokrish Sounds like it also needs the contents of what that references.
@robi perhaps there is some other way. I will try to dig a little deeper to see if there is any other solution!
-
Unrelated but I did a double take reading NFS as NFT. This forum has not been tainted yet.
-
Unrelated but I did a double take reading NFS as NFT. This forum has not been tainted yet.
-
I took a quick look at this and the proposed solution to have the dependencies properly setup are not working due to a dependency loop.
unbound requires docker
volumemount requires unbound but has to be started before dockernot quite sure how to resolve that, have to investigate further or does anyone have an idea or are there any best practices with systemd dependencies and docker container startup?
-
I took a quick look at this and the proposed solution to have the dependencies properly setup are not working due to a dependency loop.
unbound requires docker
volumemount requires unbound but has to be started before dockernot quite sure how to resolve that, have to investigate further or does anyone have an idea or are there any best practices with systemd dependencies and docker container startup?
@nebulon I don't see a loop.
You have docker already, and unbound and volume mounts can be done with volumes or binds mounts via docker cli..
https://docs.docker.com/storage/volumes/
Escape values from outer CSV parser If your volume driver accepts a comma-separated list as an option, you must escape the value from the outer CSV parser. To escape a volume-opt, surround it with double quotes (") and surround the entire mount parameter with single quotes ('). For example, the local driver accepts mount options as a comma-separated list in the o parameter. This example shows the correct way to escape the list. $ docker service create \ --mount 'type=volume,src=<VOLUME-NAME>,dst=<CONTAINER-PATH>,volume-driver=local,volume-opt=type=nfs,volume-opt=device=<nfs-server>:<nfs-path>,"volume-opt=o=addr=<nfs-address>,vers=4,soft,timeo=180,bg,tcp,rw"' --name myservice \ <IMAGE> -
@nebulon I don't see a loop.
You have docker already, and unbound and volume mounts can be done with volumes or binds mounts via docker cli..
https://docs.docker.com/storage/volumes/
Escape values from outer CSV parser If your volume driver accepts a comma-separated list as an option, you must escape the value from the outer CSV parser. To escape a volume-opt, surround it with double quotes (") and surround the entire mount parameter with single quotes ('). For example, the local driver accepts mount options as a comma-separated list in the o parameter. This example shows the correct way to escape the list. $ docker service create \ --mount 'type=volume,src=<VOLUME-NAME>,dst=<CONTAINER-PATH>,volume-driver=local,volume-opt=type=nfs,volume-opt=device=<nfs-server>:<nfs-path>,"volume-opt=o=addr=<nfs-address>,vers=4,soft,timeo=180,bg,tcp,rw"' --name myservice \ <IMAGE>@robi my explanation lacked some detail. Let me try to explain with my first morning coffee, maybe helps me sort it as well
On cloudron we have those volumes, which are essentially mounts on the host system. Those are managed by systemd and can be configured via the dashboard. Then such a volume or mounted folder essentially, can be mounted into an app. (note Cloudron volume !~ docker volume and same for mounts, the terminology is not exactly the same)
Now if the system starts up, the docker daemon will startup up the containers, at which point the host mounted folders have to be available, otherwise docker will mount empty folders into the apps. The unbound service is required for reliable volume host resolving https://git.cloudron.io/cloudron/box/-/commit/a56766ab0e7f13cbc678fee80d3cea2a86d12c71 so it has to be up an running prior to Cloudron volume mounts. All clear so far I think.
Now docker needs to be started before unbound, at least according to the unit file adjusted with https://git.cloudron.io/cloudron/box/-/commit/4d55783ed865922b155c71e8b17533fdd5d850ec this leads to the mentioned loop and systemd breaks the loop during startup by postponing host mounts in this case, according to the systemd startup logs.
Two possible solutions here. Some smart way with systemd to solve such situations, which I don't know yet, or double check why unbound requires to be run after docker. Maybe @girish can shed some light here, I can't remember why that was required back then.
-
@robi my explanation lacked some detail. Let me try to explain with my first morning coffee, maybe helps me sort it as well
On cloudron we have those volumes, which are essentially mounts on the host system. Those are managed by systemd and can be configured via the dashboard. Then such a volume or mounted folder essentially, can be mounted into an app. (note Cloudron volume !~ docker volume and same for mounts, the terminology is not exactly the same)
Now if the system starts up, the docker daemon will startup up the containers, at which point the host mounted folders have to be available, otherwise docker will mount empty folders into the apps. The unbound service is required for reliable volume host resolving https://git.cloudron.io/cloudron/box/-/commit/a56766ab0e7f13cbc678fee80d3cea2a86d12c71 so it has to be up an running prior to Cloudron volume mounts. All clear so far I think.
Now docker needs to be started before unbound, at least according to the unit file adjusted with https://git.cloudron.io/cloudron/box/-/commit/4d55783ed865922b155c71e8b17533fdd5d850ec this leads to the mentioned loop and systemd breaks the loop during startup by postponing host mounts in this case, according to the systemd startup logs.
Two possible solutions here. Some smart way with systemd to solve such situations, which I don't know yet, or double check why unbound requires to be run after docker. Maybe @girish can shed some light here, I can't remember why that was required back then.
