High Performance Back-end for Nextcloud Files

[avatar1024]

@oj Great thanks.
@staff @appdev Do you know if all those requirements can be baked in to the current Cloudron NC app or would it work best as an additional app that links with NC?

[oj]

"Syncing at a new level: 10x to 100x": With Instant Sync thanks to the High Performance Back-end for Files introduced earlier this year, and the File Packing coming to V2 of the Sync Engine, both user experience and server load will dramatically improve.

For the server, idle load will have decreased by 10x thanks to the HPB, while during syncing the number of connections is also reduced by about a factor 10!

Users will benefit from instant availability of their files, even if they need to have large numbers of small files made available locally.

Nextcloud Sync v2 will be made available as part of a desktop client release later this year, alongside the release of Nextcloud 23.

[avatar1024]

With nextcloud 23 it seems there is again more improvement on performance for files, but it probably still requires the high performance back-end.

Has anyone looked into how to enable this for NC Cloudron package?

It feels like if we don't have it we're going to fall behind other NC instances which would be a shame.

[nebulon]

@avatar1024 from my perspective, stability is more important than speed when it comes to Nextcloud features and updates. So unless their high performance backend becomes the default, I would rather not set this for our app package with the experience we have had in the past with Nextcloud.

[avatar1024]

@nebulon Sure, I agree stability is key, and we should definitely take our time with NC update!

However, HPB for files has been there since NC21 and I wasn't aware of stability issues with it. You're right that the app (notify push) is not installed by default but I think that's only because it requires extra config on the server. So I don't think we need to modify the Cloudron NC package as such but "only" create an easy way to make the appropriate change on the server. I might be wrong though.

Procedure feels fairly simple on a stand alone NC instance but I don't know how that would apply to a cloudron set-up:

(relevant part at 32:39)

[jdaviescoates]

@nebulon said in High Performance Back-end for Nextcloud Files:

stability is more important than speed

Yes, but slow speed (especially of the web ui) is also the worst thing about Nextcloud.

So anything that speeds it up would be great!

Stability is more important overall, but speed is also very important imho.

IMHO the High Performance Backend should probably be added to the Cloudron Nextcloud package (or otherwise made easy to set-up on Cloudron), but probably not just yet.

Perhaps we should wait until notify_push (as it's called on Github) is at 1.0.1? (it's currently on 0.3.0) and/ or until Client Push (as it's called in Nextcloud Apps) has a positive overall rating? (currently neutral, see https://apps.nextcloud.com/apps/notify_push ).

[jdaviescoates]

@jdaviescoates said in High Performance Back-end for Nextcloud Files:

Perhaps we should wait until notify_push (as it's called on Github) is at 1.0.1? (it's currently on 0.3.0) and/ or until Client Push (as it's called in Nextcloud Apps) has a positive overall rating? (currently neutral, see https://apps.nextcloud.com/apps/notify_push ).

But, in the meantime it'd still be nice to be able to simply be able enable it on Cloudron too.

@avatar1024 have you tried getting it to work by following those instructions?

[avatar1024]

@jdaviescoates said in High Performance Back-end for Nextcloud Files:

Perhaps we should wait until notify_push (as it's called on Github) is at 1.0.1? (it's currently on 0.3.0) and/ or until Client Push (as it's called in Nextcloud Apps) has a positive overall rating? (currently neutral, see https://apps.nextcloud.com/apps/notify_push ).

Yes but this is only the NC app side of things. People can choose to enable it or not like any other app if there is any issue with it (plenty of apps are below the 1.0.0 version). So the question is, do the changes required on the server config have an impact if the app is disabled? It didn't look like that it would be the case.

@jdaviescoates said in High Performance Back-end for Nextcloud Files:

Yes, but slow speed (especially of the web ui) is also the worst thing about Nextcloud.

Agreed

@jdaviescoates said in High Performance Back-end for Nextcloud Files:

@avatar1024 have you tried getting it to work by following those instructions?

I didn't because this tuto is for a native install (not in docker) and so I don;t know how some of these translate in the case of a docker / containerised install like the cloudron one.

If the process is easily adaptable, then activating the HBP for files could just be as simple as adding the steps on how to do it in the Cloudron NC package documentation. And there again, users can choose to go with it or not. It might not require any change to the app package itself.

[jdaviescoates]

@avatar1024 said in High Performance Back-end for Nextcloud Files:

People can choose to enable it or not like any other app if there is any issue with it (plenty of apps are below the 1.0.0 version).

Fair point.

[avatar1024]

@jdaviescoates I have now tried to just follow the basic instruction instruction on the notify push github.

What I have done:

1. Installed the app inside NC
2. run occ notify_push:setup inside the app terminal
3. The abovce generated a systemd service like

[Unit]
Description = Push daemon for Nextcloud clients

[Service]
Environment=PORT=7867
Environment=NEXTCLOUD_URL=https://files.xxxxx.net/
ExecStart=/app/data/apps/notify_push/bin/x86_64/notify_push /app/code/config/config.php
User=www-data

[Install]
WantedBy = multi-user.target

I have replaced the path to the actual path to the binary and the config file like so

[Unit]
Description = Push daemon for Nextcloud clients

[Service]
Environment=PORT=7867
Environment=NEXTCLOUD_URL=https://files.xxxxx.net/
ExecStart=/home/yellowtent/appsdata/1445e2c5-fa60-4ce9-894e-3e2a8b8dd0a0/data/apps/notify_push/bin/x86_64/notify_push /home/yellowtent/appsdata/1445e2c5-fa60-4ce9-894e-3e2a8b8dd0a0/data/config/config.php
User=www-data

[Install]
WantedBy = multi-user.target

ran sudo systemctl enable --now notify_push as indicated.

But then it says the binary is not running.... so for now it's a fail. Anyone got any idea?

[girish]

@avatar1024 said in High Performance Back-end for Nextcloud Files:

But then it says the binary is not running

Maybe systemctl status notify_push ? and journalctl -u notify_push has more info.

[avatar1024]

@girish

systemctl status notify_push gives me:

notify_push.service - Push daemon for Nextcloud clients
     Loaded: loaded (/etc/systemd/system/notify_push.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Fri 2022-04-01 09:58:44 UTC; 14h ago
    Process: 180019 ExecStart=/home/yellowtent/appsdata/1445e2c5-fa60-4ce9-894e-3e2a8b8dd0a0/data/apps/notify_push/bin/x86_64/notify_push /home/yellowtent/appsdata/1445e2c5-fa60-4ce9-894e-3e2a8b8dd0a0/data/config/config.php (code=exited, status=1/FAILURE)
   Main PID: 180019 (code=exited, status=1/FAILURE)

Apr 01 09:58:44 v2202201132182176931 notify_push[180019]:    0: Failed to connect to Nextcloud database
Apr 01 09:58:44 v2202201132182176931 notify_push[180019]:    1: error communicating with the server: failed to lookup address information: Name does not resolve
Apr 01 09:58:44 v2202201132182176931 notify_push[180019]:    2: failed to lookup address information: Name does not resolve
Apr 01 09:58:44 v2202201132182176931 notify_push[180019]: Location:
Apr 01 09:58:44 v2202201132182176931 notify_push[180019]:    /tmp/krankerl.UyibIgt7EFoA/notify_push/src/storage_mapping.rs:58
Apr 01 09:58:44 v2202201132182176931 notify_push[180019]: Backtrace omitted.
Apr 01 09:58:44 v2202201132182176931 notify_push[180019]: Run with RUST_BACKTRACE=1 environment variable to display it.
Apr 01 09:58:44 v2202201132182176931 notify_push[180019]: Run with RUST_BACKTRACE=full to include source snippets.
Apr 01 09:58:44 v2202201132182176931 systemd[1]: notify_push.service: Main process exited, code=exited, status=1/FAILURE
Apr 01 09:58:44 v2202201132182176931 systemd[1]: notify_push.service: Failed with result 'exit-code'.

And journalctl -u notify_push

-- Logs begin at Mon 2022-03-28 21:45:49 UTC, end at Sat 2022-04-02 00:34:23 UTC. --
Apr 01 09:33:43 v2202201132182176931 systemd[1]: Started Push daemon for Nextcloud clients.
Apr 01 09:33:43 v2202201132182176931 notify_push[176600]: Error:
Apr 01 09:33:43 v2202201132182176931 notify_push[176600]:    0: Failed to parse config
Apr 01 09:33:43 v2202201132182176931 notify_push[176600]:    1: Failed to parse nextcloud config
Apr 01 09:33:43 v2202201132182176931 notify_push[176600]:    2: Failed to read config file
Apr 01 09:33:43 v2202201132182176931 notify_push[176600]: Location:
Apr 01 09:33:43 v2202201132182176931 notify_push[176600]:    src/config/nc.rs:8
Apr 01 09:33:43 v2202201132182176931 notify_push[176600]: Backtrace omitted.
Apr 01 09:33:43 v2202201132182176931 notify_push[176600]: Run with RUST_BACKTRACE=1 environment variable to display it.
Apr 01 09:33:43 v2202201132182176931 notify_push[176600]: Run with RUST_BACKTRACE=full to include source snippets.
Apr 01 09:33:43 v2202201132182176931 systemd[1]: notify_push.service: Main process exited, code=exited, status=1/FAILURE
Apr 01 09:33:43 v2202201132182176931 systemd[1]: notify_push.service: Failed with result 'exit-code'.
Apr 01 09:34:44 v2202201132182176931 systemd[1]: Started Push daemon for Nextcloud clients.
Apr 01 09:34:44 v2202201132182176931 notify_push[176680]: Error:
Apr 01 09:34:44 v2202201132182176931 notify_push[176680]:    0: Failed to parse config
Apr 01 09:34:44 v2202201132182176931 notify_push[176680]:    1: Failed to parse nextcloud config
Apr 01 09:34:44 v2202201132182176931 notify_push[176680]:    2: Failed to read config file
Apr 01 09:34:44 v2202201132182176931 notify_push[176680]: Location:
Apr 01 09:34:44 v2202201132182176931 notify_push[176680]:    src/config/nc.rs:8
Apr 01 09:34:44 v2202201132182176931 notify_push[176680]: Backtrace omitted.
Apr 01 09:34:44 v2202201132182176931 notify_push[176680]: Run with RUST_BACKTRACE=1 environment variable to display it.
Apr 01 09:34:44 v2202201132182176931 notify_push[176680]: Run with RUST_BACKTRACE=full to include source snippets.
Apr 01 09:34:44 v2202201132182176931 systemd[1]: notify_push.service: Main process exited, code=exited, status=1/FAILURE
Apr 01 09:34:44 v2202201132182176931 systemd[1]: notify_push.service: Failed with result 'exit-code'.
Apr 01 09:35:05 v2202201132182176931 notify_push[176786]: Location:
Apr 01 09:35:05 v2202201132182176931 notify_push[176786]:    src/config/nc.rs:8
Apr 01 09:35:05 v2202201132182176931 notify_push[176786]: Backtrace omitted.
Apr 01 09:35:05 v2202201132182176931 notify_push[176786]: Run with RUST_BACKTRACE=1 environment variable to display it.
Apr 01 09:35:05 v2202201132182176931 notify_push[176786]: Run with RUST_BACKTRACE=full to include source snippets.
Apr 01 09:35:05 v2202201132182176931 systemd[1]: notify_push.service: Main process exited, code=exited, status=1/FAILURE
Apr 01 09:35:05 v2202201132182176931 systemd[1]: notify_push.service: Failed with result 'exit-code'.
Apr 01 09:54:28 v2202201132182176931 systemd[1]: Started Push daemon for Nextcloud clients.
Apr 01 09:54:28 v2202201132182176931 notify_push[179480]: Error:
Apr 01 09:54:28 v2202201132182176931 notify_push[179480]:    0: Failed to connect to Nextcloud database
Apr 01 09:54:28 v2202201132182176931 notify_push[179480]:    1: error communicating with the server: failed to lookup address information: Name does not resolve
Apr 01 09:54:28 v2202201132182176931 notify_push[179480]:    2: failed to lookup address information: Name does not resolve
Apr 01 09:54:28 v2202201132182176931 notify_push[179480]: Location:
Apr 01 09:54:28 v2202201132182176931 notify_push[179480]:    /tmp/krankerl.UyibIgt7EFoA/notify_push/src/storage_mapping.rs:58
Apr 01 09:54:28 v2202201132182176931 notify_push[179480]: Backtrace omitted.
Apr 01 09:54:28 v2202201132182176931 notify_push[179480]: Run with RUST_BACKTRACE=1 environment variable to display it.
Apr 01 09:54:28 v2202201132182176931 notify_push[179480]: Run with RUST_BACKTRACE=full to include source snippets.
Apr 01 09:54:28 v2202201132182176931 systemd[1]: notify_push.service: Main process exited, code=exited, status=1/FAILURE
Apr 01 09:54:28 v2202201132182176931 systemd[1]: notify_push.service: Failed with result 'exit-code'.
Apr 01 09:58:44 v2202201132182176931 systemd[1]: Started Push daemon for Nextcloud clients.
Apr 01 09:58:44 v2202201132182176931 notify_push[180019]: Error:
Apr 01 09:58:44 v2202201132182176931 notify_push[180019]:    0: Failed to connect to Nextcloud database
Apr 01 09:58:44 v2202201132182176931 notify_push[180019]:    1: error communicating with the server: failed to lookup address information: Name does not resolve
Apr 01 09:58:44 v2202201132182176931 notify_push[180019]:    2: failed to lookup address information: Name does not resolve
Apr 01 09:58:44 v2202201132182176931 notify_push[180019]: Location:
Apr 01 09:58:44 v2202201132182176931 notify_push[180019]:    /tmp/krankerl.UyibIgt7EFoA/notify_push/src/storage_mapping.rs:58
Apr 01 09:58:44 v2202201132182176931 notify_push[180019]: Backtrace omitted.
Apr 01 09:58:44 v2202201132182176931 notify_push[180019]: Run with RUST_BACKTRACE=1 environment variable to display it.
Apr 01 09:58:44 v2202201132182176931 notify_push[180019]: Run with RUST_BACKTRACE=full to include source snippets.
Apr 01 09:58:44 v2202201132182176931 systemd[1]: notify_push.service: Main process exited, code=exited, status=1/FAILURE
Apr 01 09:58:44 v2202201132182176931 systemd[1]: notify_push.service: Failed with result 'exit-code'.

[girish]

@avatar1024 said in High Performance Back-end for Nextcloud Files:

/home/yellowtent/appsdata/1445e2c5-fa60-4ce9-894e-3e2a8b8dd0a0/data/apps/notify_push/bin/x86_64/notify_push /home/yellowtent/appsdata/1445e2c5-fa60-4ce9-894e-3e2a8b8dd0a0/data/config/config.php

You should try running that command standalone first. Looks like that doesn't work. (It's unable to locate configs or something?)

[eganonoa]

@girish @avatar1024 I was trying to follow this just now and am confused how you even got that far. It seems like @avatar1024 was able to create the systemd unit file, but I can't get anywhere with that because it's a readonly file system. Am I missing something here? Is there a way to place a new systemd unit file within a cloudron docker container? [Otherwise, I'm wondering whether @avatar1024 put the unit file into the base of the cloudron server, and not the docker container, which would explain why it doesn't find the relevant binary and config].

It would be really useful to figure this out and understand where and how I must put a systemd unit file within a read-only container. Or if not that to otherwise know how to create a binary that won't get overwritten on restart, but would rather autostart (see e.g. how linuxserver containers do this: https://github.com/linuxserver/docker-nextcloud/issues/194)

Beyond that, it looks like I have been able to get notify push to work by simply calling the binary itself within the nextcloud container. The steps I took were as follows (following this guide https://github.com/nextcloud/notify_push) :

1. Install and enable notify_push app in Nextcloud
2. In the terminal of the nextcloud container directly run the binary with the relevant variables (./app/data/apps/notify_push/bin/x86_64/notify_push --allow-self-signed --nextcloud-url https://[Nextcloud URL] --port 7867 -- /app/data/config/config.php)
3. In the overall cloudron server edit the relevant nginx config for the Nextcloud container to add the location block as per the github readme (changing the IP address to that of the container) and reloading nginx
4. Open another terminal for the nextcloud container and add the relevant IP address to the trusted proxies array in the Nextcloud config.php
5. Enable the app (occ app:enable notify_push)
6. set the url of the push server (occ notify_push:setup https://url/push)

Once done I get a nice confirmation that everything is working with green ticks for all of the relevant parts:

✓ redis is configured
✓ push server is receiving redis messages
✓ push server can load mount info from database
✓ push server can connect to the Nextcloud server
✓ push server is a trusted proxy
✓ push server is running the same version as the app

And when I run occ notify_push:metrics it gives me metrics that appear to confirm that everything is working well.

So, as I say, it all seems quite possible, though nothing is going to survive a reboot and I'd love to find a way to at least autostart running the binary if not setting it all up as a systemd service within the docker container. Any help would be much appreciated.

[girish]

@eganonoa said in High Performance Back-end for Nextcloud Files:

@girish @avatar1024 I was trying to follow this just now and am confused how you even got that far. It seems like @avatar1024 was able to create the systemd unit file, but I can't get anywhere with that because it's a readonly file system. Am I missing something here? Is there a way to place a new systemd unit file within a cloudron docker container? [Otherwise, I'm wondering whether @avatar1024 put the unit file into the base of the cloudron server, and not the docker container, which would explain why it doesn't find the relevant binary and config].

It's basically what you guessed! Since the systemd conf cannot be inside docker, he put it on the host. And since it's on the host, the paths were wrong.

[girish]

@eganonoa said in High Performance Back-end for Nextcloud Files:

So, as I say, it all seems quite possible, though nothing is going to survive a reboot and I'd love to find a way to at least autostart running the binary if not setting it all up as a systemd service within the docker container. Any help would be much appreciated.

I was investigating how we can support such a system via app cron. I found that there is an extension to cron named @reboot which is exactly what we need here. (i.e this keyword will translate to run on restart of an app)

[eganonoa]

@girish I was just now wondering whether cron could do something like this! That seems ideal. I'm going to see if I can make it work.

[girish]

I have implemented the patterns for next release (7.2.1). You can now put things like @service /some/service.sh in the crontab.

[eganonoa]

@girish Awesome! I think this is pretty close now. We are going to need a script along the lines of the one in here but with the relevant flags and options included. We can place that in the app/data/config folder and it won't get overwritten.

Potentially this could be useful for a variety of things, much as the linuxserver folks appear to have found.

[Edit: your cron change being the thing that would be "useful for a variety of things"]

[eganonoa]

@girish Just to walk through where I think things stand on this and where some addition input might be needed even after the cron changes allow for autostart and to help anyone who comes here in the interim.

1. Because of the read-only file system, the HBP files cannot be set up as a background daemon run by the init system as detailed in the main guide: https://github.com/nextcloud/notify_push.

2. Nonetheless, HBP files can be made to work (with manual intervention after a reboot) as follows (slight changes/clarifications from above):

• Install (but do not enable) the Client Push app in Nextcloud
• Edit Nginx Find the local IP of your nextcloud container. In the main cloudron server navigate to /etc/nginx/applications and open the .conf for your nextcloud container. Add the location block lhere for push into that file, but changing the IP to the local IP of your nextcloud container. Reload nginx.
• Run the notify push binary. As currently there is no way to run it as a background task (see below) my recommendation here is to enter the container via the main server and use tmux/screen so that you can run the binary and attach the session. Run the following from the /app/code directory: ./apps/notify_push/bin/x86_64/notify_push --allow-self-signed --nextcloud-url http://[Local IP] --port 7867 -- /app/data/config/config.php. Note http, not https and local IP and not the nextcloud URL. The standard way is to use https://[nextcloud url] and you can try it but I believe you will ultimately have trouble with the trusted proxy settings and will need to bypass the proxy server. This appears to be a fairly common issue. Detach the tmux session so that the binary keep. running.
• Update the Nextcloud Config adding the local IP address from above into the trusted proxy array.
• Enable the Notify Push App: occ app:enable notify_push
• Setup the HBP Files: occ notify_push:setup https://[nextcloud url]/push
• Check if everything works: First you should get only green ticks all the way down as it verifies if things are working. Then refresh any browser sessions and close and restart any desktop clients. After that run occ notify_push:metrics and there you should see that there are a number of active connections and requests being made. Then a really nice way to check if it is all working is to go into Talk and have someone (or another device) call you. You should see an instant notification of the call on your desktop (if you use a client) or in your browser vs before when you would have a 15 second delay.

3. Currently this all has to be done manually and will need to be redone whenever you restart the container (with exception of Nginx, which might only be overwritten when the server is rebooted). @girish has fixed things so that from Cloudron version 7.2.1 we can run cron jobs on reboot. So now we need to get things lined up for that. The solution is partially outlined here. We need to put a script in a folder that isn't overwritten, such as the nextcloud config directory and have it run on reboot. A working script would look include:

#!/bin/sh
NEXTCLOUD_BASE="/app/code"
NOTIFY_PUSH_BIN="$NEXTCLOUD_BASE/apps/notify_push/bin/x86_64/notify_push"
if [ -f "$NOTIFY_PUSH_BIN" ]; then
  "$NOTIFY_PUSH_BIN" --allow-self-signed --nextcloud-url http://[Local IP] --port 7867 "$NEXTCLOUD_BASE/config/config.php"
fi

But that will not be sufficient. This will need to be preceded by something that would add the local IP into the config.php file (because this will be overwritten), and followed by the occ enable and notify_push commands in order to get this to work.

My assumption is that this could all be put into one executable and then we can simply have instructions on the cloudron man pages for anyone who wants to enable the HBP Files. Any help with writing this would be much appreciated.