High Performance Back-end for Nextcloud Files

girish

@avatar1024 said in High Performance Back-end for Nextcloud Files:

But then it says the binary is not running

Maybe systemctl status notify_push ? and journalctl -u notify_push has more info.

avatar1024

systemctl status notify_push gives me:

notify_push.service - Push daemon for Nextcloud clients
     Loaded: loaded (/etc/systemd/system/notify_push.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Fri 2022-04-01 09:58:44 UTC; 14h ago
    Process: 180019 ExecStart=/home/yellowtent/appsdata/1445e2c5-fa60-4ce9-894e-3e2a8b8dd0a0/data/apps/notify_push/bin/x86_64/notify_push /home/yellowtent/appsdata/1445e2c5-fa60-4ce9-894e-3e2a8b8dd0a0/data/config/config.php (code=exited, status=1/FAILURE)
   Main PID: 180019 (code=exited, status=1/FAILURE)

Apr 01 09:58:44 v2202201132182176931 notify_push[180019]:    0: Failed to connect to Nextcloud database
Apr 01 09:58:44 v2202201132182176931 notify_push[180019]:    1: error communicating with the server: failed to lookup address information: Name does not resolve
Apr 01 09:58:44 v2202201132182176931 notify_push[180019]:    2: failed to lookup address information: Name does not resolve
Apr 01 09:58:44 v2202201132182176931 notify_push[180019]: Location:
Apr 01 09:58:44 v2202201132182176931 notify_push[180019]:    /tmp/krankerl.UyibIgt7EFoA/notify_push/src/storage_mapping.rs:58
Apr 01 09:58:44 v2202201132182176931 notify_push[180019]: Backtrace omitted.
Apr 01 09:58:44 v2202201132182176931 notify_push[180019]: Run with RUST_BACKTRACE=1 environment variable to display it.
Apr 01 09:58:44 v2202201132182176931 notify_push[180019]: Run with RUST_BACKTRACE=full to include source snippets.
Apr 01 09:58:44 v2202201132182176931 systemd[1]: notify_push.service: Main process exited, code=exited, status=1/FAILURE
Apr 01 09:58:44 v2202201132182176931 systemd[1]: notify_push.service: Failed with result 'exit-code'.

And journalctl -u notify_push

-- Logs begin at Mon 2022-03-28 21:45:49 UTC, end at Sat 2022-04-02 00:34:23 UTC. --
Apr 01 09:33:43 v2202201132182176931 systemd[1]: Started Push daemon for Nextcloud clients.
Apr 01 09:33:43 v2202201132182176931 notify_push[176600]: Error:
Apr 01 09:33:43 v2202201132182176931 notify_push[176600]:    0: Failed to parse config
Apr 01 09:33:43 v2202201132182176931 notify_push[176600]:    1: Failed to parse nextcloud config
Apr 01 09:33:43 v2202201132182176931 notify_push[176600]:    2: Failed to read config file
Apr 01 09:33:43 v2202201132182176931 notify_push[176600]: Location:
Apr 01 09:33:43 v2202201132182176931 notify_push[176600]:    src/config/nc.rs:8
Apr 01 09:33:43 v2202201132182176931 notify_push[176600]: Backtrace omitted.
Apr 01 09:33:43 v2202201132182176931 notify_push[176600]: Run with RUST_BACKTRACE=1 environment variable to display it.
Apr 01 09:33:43 v2202201132182176931 notify_push[176600]: Run with RUST_BACKTRACE=full to include source snippets.
Apr 01 09:33:43 v2202201132182176931 systemd[1]: notify_push.service: Main process exited, code=exited, status=1/FAILURE
Apr 01 09:33:43 v2202201132182176931 systemd[1]: notify_push.service: Failed with result 'exit-code'.
Apr 01 09:34:44 v2202201132182176931 systemd[1]: Started Push daemon for Nextcloud clients.
Apr 01 09:34:44 v2202201132182176931 notify_push[176680]: Error:
Apr 01 09:34:44 v2202201132182176931 notify_push[176680]:    0: Failed to parse config
Apr 01 09:34:44 v2202201132182176931 notify_push[176680]:    1: Failed to parse nextcloud config
Apr 01 09:34:44 v2202201132182176931 notify_push[176680]:    2: Failed to read config file
Apr 01 09:34:44 v2202201132182176931 notify_push[176680]: Location:
Apr 01 09:34:44 v2202201132182176931 notify_push[176680]:    src/config/nc.rs:8
Apr 01 09:34:44 v2202201132182176931 notify_push[176680]: Backtrace omitted.
Apr 01 09:34:44 v2202201132182176931 notify_push[176680]: Run with RUST_BACKTRACE=1 environment variable to display it.
Apr 01 09:34:44 v2202201132182176931 notify_push[176680]: Run with RUST_BACKTRACE=full to include source snippets.
Apr 01 09:34:44 v2202201132182176931 systemd[1]: notify_push.service: Main process exited, code=exited, status=1/FAILURE
Apr 01 09:34:44 v2202201132182176931 systemd[1]: notify_push.service: Failed with result 'exit-code'.
Apr 01 09:35:05 v2202201132182176931 notify_push[176786]: Location:
Apr 01 09:35:05 v2202201132182176931 notify_push[176786]:    src/config/nc.rs:8
Apr 01 09:35:05 v2202201132182176931 notify_push[176786]: Backtrace omitted.
Apr 01 09:35:05 v2202201132182176931 notify_push[176786]: Run with RUST_BACKTRACE=1 environment variable to display it.
Apr 01 09:35:05 v2202201132182176931 notify_push[176786]: Run with RUST_BACKTRACE=full to include source snippets.
Apr 01 09:35:05 v2202201132182176931 systemd[1]: notify_push.service: Main process exited, code=exited, status=1/FAILURE
Apr 01 09:35:05 v2202201132182176931 systemd[1]: notify_push.service: Failed with result 'exit-code'.
Apr 01 09:54:28 v2202201132182176931 systemd[1]: Started Push daemon for Nextcloud clients.
Apr 01 09:54:28 v2202201132182176931 notify_push[179480]: Error:
Apr 01 09:54:28 v2202201132182176931 notify_push[179480]:    0: Failed to connect to Nextcloud database
Apr 01 09:54:28 v2202201132182176931 notify_push[179480]:    1: error communicating with the server: failed to lookup address information: Name does not resolve
Apr 01 09:54:28 v2202201132182176931 notify_push[179480]:    2: failed to lookup address information: Name does not resolve
Apr 01 09:54:28 v2202201132182176931 notify_push[179480]: Location:
Apr 01 09:54:28 v2202201132182176931 notify_push[179480]:    /tmp/krankerl.UyibIgt7EFoA/notify_push/src/storage_mapping.rs:58
Apr 01 09:54:28 v2202201132182176931 notify_push[179480]: Backtrace omitted.
Apr 01 09:54:28 v2202201132182176931 notify_push[179480]: Run with RUST_BACKTRACE=1 environment variable to display it.
Apr 01 09:54:28 v2202201132182176931 notify_push[179480]: Run with RUST_BACKTRACE=full to include source snippets.
Apr 01 09:54:28 v2202201132182176931 systemd[1]: notify_push.service: Main process exited, code=exited, status=1/FAILURE
Apr 01 09:54:28 v2202201132182176931 systemd[1]: notify_push.service: Failed with result 'exit-code'.
Apr 01 09:58:44 v2202201132182176931 systemd[1]: Started Push daemon for Nextcloud clients.
Apr 01 09:58:44 v2202201132182176931 notify_push[180019]: Error:
Apr 01 09:58:44 v2202201132182176931 notify_push[180019]:    0: Failed to connect to Nextcloud database
Apr 01 09:58:44 v2202201132182176931 notify_push[180019]:    1: error communicating with the server: failed to lookup address information: Name does not resolve
Apr 01 09:58:44 v2202201132182176931 notify_push[180019]:    2: failed to lookup address information: Name does not resolve
Apr 01 09:58:44 v2202201132182176931 notify_push[180019]: Location:
Apr 01 09:58:44 v2202201132182176931 notify_push[180019]:    /tmp/krankerl.UyibIgt7EFoA/notify_push/src/storage_mapping.rs:58
Apr 01 09:58:44 v2202201132182176931 notify_push[180019]: Backtrace omitted.
Apr 01 09:58:44 v2202201132182176931 notify_push[180019]: Run with RUST_BACKTRACE=1 environment variable to display it.
Apr 01 09:58:44 v2202201132182176931 notify_push[180019]: Run with RUST_BACKTRACE=full to include source snippets.
Apr 01 09:58:44 v2202201132182176931 systemd[1]: notify_push.service: Main process exited, code=exited, status=1/FAILURE
Apr 01 09:58:44 v2202201132182176931 systemd[1]: notify_push.service: Failed with result 'exit-code'.

girish

@avatar1024 said in High Performance Back-end for Nextcloud Files:

/home/yellowtent/appsdata/1445e2c5-fa60-4ce9-894e-3e2a8b8dd0a0/data/apps/notify_push/bin/x86_64/notify_push /home/yellowtent/appsdata/1445e2c5-fa60-4ce9-894e-3e2a8b8dd0a0/data/config/config.php

You should try running that command standalone first. Looks like that doesn't work. (It's unable to locate configs or something?)

eganonoa

@girish @avatar1024 I was trying to follow this just now and am confused how you even got that far. It seems like @avatar1024 was able to create the systemd unit file, but I can't get anywhere with that because it's a readonly file system. Am I missing something here? Is there a way to place a new systemd unit file within a cloudron docker container? [Otherwise, I'm wondering whether @avatar1024 put the unit file into the base of the cloudron server, and not the docker container, which would explain why it doesn't find the relevant binary and config].

It would be really useful to figure this out and understand where and how I must put a systemd unit file within a read-only container. Or if not that to otherwise know how to create a binary that won't get overwritten on restart, but would rather autostart (see e.g. how linuxserver containers do this: https://github.com/linuxserver/docker-nextcloud/issues/194)

Beyond that, it looks like I have been able to get notify push to work by simply calling the binary itself within the nextcloud container. The steps I took were as follows (following this guide https://github.com/nextcloud/notify_push) :

Install and enable notify_push app in Nextcloud
In the terminal of the nextcloud container directly run the binary with the relevant variables (./app/data/apps/notify_push/bin/x86_64/notify_push --allow-self-signed --nextcloud-url https://[Nextcloud URL] --port 7867 -- /app/data/config/config.php)
In the overall cloudron server edit the relevant nginx config for the Nextcloud container to add the location block as per the github readme (changing the IP address to that of the container) and reloading nginx
Open another terminal for the nextcloud container and add the relevant IP address to the trusted proxies array in the Nextcloud config.php
Enable the app (occ app:enable notify_push)
set the url of the push server (occ notify_push:setup https://url/push)

Once done I get a nice confirmation that everything is working with green ticks for all of the relevant parts:

✓ redis is configured
✓ push server is receiving redis messages
✓ push server can load mount info from database
✓ push server can connect to the Nextcloud server
✓ push server is a trusted proxy
✓ push server is running the same version as the app

And when I run occ notify_push:metrics it gives me metrics that appear to confirm that everything is working well.

So, as I say, it all seems quite possible, though nothing is going to survive a reboot and I'd love to find a way to at least autostart running the binary if not setting it all up as a systemd service within the docker container. Any help would be much appreciated.

girish

@eganonoa said in High Performance Back-end for Nextcloud Files:

@girish @avatar1024 I was trying to follow this just now and am confused how you even got that far. It seems like @avatar1024 was able to create the systemd unit file, but I can't get anywhere with that because it's a readonly file system. Am I missing something here? Is there a way to place a new systemd unit file within a cloudron docker container? [Otherwise, I'm wondering whether @avatar1024 put the unit file into the base of the cloudron server, and not the docker container, which would explain why it doesn't find the relevant binary and config].

It's basically what you guessed! Since the systemd conf cannot be inside docker, he put it on the host. And since it's on the host, the paths were wrong.

girish

@eganonoa said in High Performance Back-end for Nextcloud Files:

So, as I say, it all seems quite possible, though nothing is going to survive a reboot and I'd love to find a way to at least autostart running the binary if not setting it all up as a systemd service within the docker container. Any help would be much appreciated.

I was investigating how we can support such a system via app cron. I found that there is an extension to cron named @reboot which is exactly what we need here. (i.e this keyword will translate to run on restart of an app)

eganonoa

@girish I was just now wondering whether cron could do something like this! That seems ideal. I'm going to see if I can make it work.

girish

I have implemented the patterns for next release (7.2.1). You can now put things like @service /some/service.sh in the crontab.

eganonoa

@girish Awesome! I think this is pretty close now. We are going to need a script along the lines of the one in here but with the relevant flags and options included. We can place that in the app/data/config folder and it won't get overwritten.

Potentially this could be useful for a variety of things, much as the linuxserver folks appear to have found.

[Edit: your cron change being the thing that would be "useful for a variety of things"]

eganonoa

@girish Just to walk through where I think things stand on this and where some addition input might be needed even after the cron changes allow for autostart and to help anyone who comes here in the interim.

Because of the read-only file system, the HBP files cannot be set up as a background daemon run by the init system as detailed in the main guide: https://github.com/nextcloud/notify_push.
Nonetheless, HBP files can be made to work (with manual intervention after a reboot) as follows (slight changes/clarifications from above):

Install (but do not enable) the Client Push app in Nextcloud
Edit Nginx Find the local IP of your nextcloud container. In the main cloudron server navigate to /etc/nginx/applications and open the .conf for your nextcloud container. Add the location block lhere for push into that file, but changing the IP to the local IP of your nextcloud container. Reload nginx.
Run the notify push binary. As currently there is no way to run it as a background task (see below) my recommendation here is to enter the container via the main server and use tmux/screen so that you can run the binary and attach the session. Run the following from the /app/code directory: ./apps/notify_push/bin/x86_64/notify_push --allow-self-signed --nextcloud-url http://[Local IP] --port 7867 -- /app/data/config/config.php. Note http, not https and local IP and not the nextcloud URL. The standard way is to use https://[nextcloud url] and you can try it but I believe you will ultimately have trouble with the trusted proxy settings and will need to bypass the proxy server. This appears to be a fairly common issue. Detach the tmux session so that the binary keep. running.
Update the Nextcloud Config adding the local IP address from above into the trusted proxy array.
Enable the Notify Push App: occ app:enable notify_push
Setup the HBP Files: occ notify_push:setup https://[nextcloud url]/push
Check if everything works: First you should get only green ticks all the way down as it verifies if things are working. Then refresh any browser sessions and close and restart any desktop clients. After that run occ notify_push:metrics and there you should see that there are a number of active connections and requests being made. Then a really nice way to check if it is all working is to go into Talk and have someone (or another device) call you. You should see an instant notification of the call on your desktop (if you use a client) or in your browser vs before when you would have a 15 second delay.

Currently this all has to be done manually and will need to be redone whenever you restart the container (with exception of Nginx, which might only be overwritten when the server is rebooted). @girish has fixed things so that from Cloudron version 7.2.1 we can run cron jobs on reboot. So now we need to get things lined up for that. The solution is partially outlined here. We need to put a script in a folder that isn't overwritten, such as the nextcloud config directory and have it run on reboot. A working script would look include:

#!/bin/sh
NEXTCLOUD_BASE="/app/code"
NOTIFY_PUSH_BIN="$NEXTCLOUD_BASE/apps/notify_push/bin/x86_64/notify_push"
if [ -f "$NOTIFY_PUSH_BIN" ]; then
  "$NOTIFY_PUSH_BIN" --allow-self-signed --nextcloud-url http://[Local IP] --port 7867 "$NEXTCLOUD_BASE/config/config.php"
fi

But that will not be sufficient. This will need to be preceded by something that would add the local IP into the config.php file (because this will be overwritten), and followed by the occ enable and notify_push commands in order to get this to work.

My assumption is that this could all be put into one executable and then we can simply have instructions on the cloudron man pages for anyone who wants to enable the HBP Files. Any help with writing this would be much appreciated.

eganonoa

Noticed that the app-level nginx conf is getting overwritten daily, breaking the backend. Am trying the fix suggested by @girish here.

eganonoa

Looks to be working. HBP Files has been running a day and still is.

So now I think it's about turning to the script to run on reboot.

eganonoa

@girish it looks like this simple script works:

#!/bin/sh
NEXTCLOUD_BASE="/app/code"
NOTIFY_PUSH_BIN="$NEXTCLOUD_BASE/apps/notify_push/bin/x86_64/notify_push"
if [ -f "$NOTIFY_PUSH_BIN" ]; then
sed -i "/[Trusted_Proxy_IP_0_in_config.php]/a \ \ \ \ 1 => '[LOCAL_IP]'," $NEXTCLOUD_BASE/config/config.php
sudo -u www-data php $NEXTCLOUD_BASE/occ app:disable notify_push
  "$NOTIFY_PUSH_BIN" --allow-self-signed --nextcloud-url http://[LOCAL_IP] --port 7867 "$NEXTCLOUD_BASE/config/config.php" &
sudo -u www-data php $NEXTCLOUD_BASE/occ app:enable notify_push
sudo -u www-data php $NEXTCLOUD_BASE/occ notify_push:setup https://[NEXTCLOUD_URL]/push
fi

Save it into /app/code/config in the nextcloud container, make it executable and run it. Runs the notify_push command in the background, so no need of the tmux hackyness from my earlier post. It makes the set up simpler, requiring just installing the app in the Nextcloud portal, changing the nginx config (as per the prior post) and creating and running this script. The script adds the necessary trusted proxy before running the other commands, getting around the fact that it gets overwritten on restart of the container.

So now it's just about testing the cron concept you have and testing this out on others' systems to see if it works for all. Any feedback from others much appreciated.

girish

@eganonoa Looks great! The cron stuff should be released with the next release in the coming 2-3 days.

necrevistonnezr

@eganonoa said in High Performance Back-end for Nextcloud Files:

Save it into /app/code/config in the nextcloud container, make it executable and run it.

…. but you can’t do that as a Cloudron admin or was that directed at @girish ?

Have you noticed performance improvements - in particular against the improvements coming with NC 24?

eganonoa

@girish Awesome! Thank you. Put it into production today and can see all the team on it with no problems. Looking good.

One thing that would be very helpful would be if the default Nextcloud nginx.conf could be set to include the block from the guide, i.e.:

location ^~ /push/ {
    proxy_pass http://[LOCAL_IP]:7867/;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "Upgrade";
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}

Presumably that won't cause an issue for installs that don't want the HPB Files (though please correct me if I have that wrong). But it would massively simplify things for those that do, guaranteeing no overwrite of the nginx config and allowing for a very simple instruction to be added in the manual (i.e. find local IP, modify script and make executable, run executable to make sure everything is working, add to cron on reboot).

eganonoa

@necrevistonnezr

"…. but you can’t do that as a Cloudron admin or was that directed at @girish ?"

Sorry, I don't quite understand what you are meaning here. If you mean, can a Cloudron customer add something into /app/code/config and have that survive a reboot, surprisingly the answer is "yes". That folder isn't read only and whatever is put in there survives a reboot.

"Have you noticed performance improvements - in particular against the improvements coming with NC 24?"

Keep in mind that we are only a small team (10 people). They say that the speed increase will only be marginal for a team my size.

Having said that, yes! We're noticing a big increase in speed on two fronts:

(1) File Sync: Last Tuesday I was giving a new person a walk through Nextcloud. I was showing how sync works by creating a new file on the cloud and showing how that shows up on the desktop. It took forever for it to happen and was a bit embarrassing given all I had said in advance. Now it is as close to instant as you can get. Create new document online and immediately see it reflected on the desktop. Very nice.

(2) Notifications These are nearly immediately coming through. Start a call and the "join call" button shows up on the desktop within 2-3 seconds. Send a direct message to someone and it seems to nearly instantly arrive.

As it happens, I did this before upgrading to NC 24, but also wanted to see how things work with it, so did an upgrade. I don't see NC 24 adding that much additional speed to those two things vs the HPB files. But NC 24 is clearly a positive upgrade, just from a day of use. Getting positive comments from the team on the browser performance (inc. people in places with much worse internet than me). Some nice additions to certain apps (e.g. mail, office, text editor, all noticeably improved, file locks on desktop files being edited online). And if you are on Windows and presumably Mac you can reply to directed messages from the desktop client (sadly we use linux desktops so don't have that).

All-in-all I'd say we had a solid upgrade with Nextcloud between figuring out the HPB Files (that was the biggest) and NC 24 (a smaller but meaningful upgrade) (in addition to recently adding the high-performance back-end for talk). Would love to see what happens if/when Cloudron updates to PhP 8. But irrespective Nextcloud is getting closer and closer to being an all-in-one solution for us, which is pretty nice. Not there yet, but I can imagine it now that things are snappier.

jdaviescoates

@eganonoa that all sounds great, thank you for working this out. I've not closely followed the whole thread, but it sounds like we're getting close to be able to say: it you want to enable Nextcloud's High Performance Back-end for Nextcloud Files do this: ....

I'd really love to be able to use the HPB with my Cloudron Nextcloud!

girish

@eganonoa I have to read more on that this service is , but we have to use the apache config - https://github.com/nextcloud/notify_push#apache .

BTW, why is this just not part of nextcloud itself? Seems complicated having to maintain this as a parallel system outside of nextcloud.

eganonoa

@girish

"I have to read more on that this service is , but we have to use the apache config - https://github.com/nextcloud/notify_push#apache".

The way I did it was to set it up via the nginx config in the main cloudron server and then use the direct IP when running the binary in order to bypass the second proxy. So I didn't touch the apache config in the Nextcloud app (I understand it goes through both the Nginx proxy on cloudron and the apache on the Nextcloud container, right?). As of now everything works perfectly and if you don't add the block into Nginx it will fail. That's why I think it would be useful to automatically have that block in there (it redirects [NEXTCLOUD URL]/push).

"BTW, why is this just not part of nextcloud itself? Seems complicated having to maintain this as a parallel system outside of nextcloud."

Indeed it does. Given that it is just a simple service in the end, it doesn't make a whole lot of sense. Probably because it's ultimately all about desktop sync speed (with notifications a bonus) they keep it out of the main one for those who only use the web-interface or to keep overheads lower for people who either don't care about speed or would rather not add overhead to their nextcloud offerings (I'm thinking of those who might be providing it for free or for very low cost).

Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.

Cloudron Forum

High Performance Back-end for Nextcloud Files