High Performance Back-end for Nextcloud Files

[jdaviescoates]

There is already an App Wishlist post for the Nextcloud Talk High Performance Back-end but I figured I'd add one for the High Performance Back-end for Nextcloud Files too.

https://nextcloud.com/blog/nextcloud-faster-than-ever-introducing-files-high-performance-back-end/
https://github.com/nextcloud/notify_push

It'd be great to have this on Cloudron!

As @oj said in Nextcloud 21 is out:

@girish @nebulon For users who are using Nextcloud + Nextcloud Talk for remote, collaborative work, the two Nextcloud HPB's (the Spreed SFU/ signalling HPB + this new notifications HPB binary) are very important.

My largest (non-Cloudron) Nextcloud has 22 remote users; the Spreed SFU (available as-a-service for 40 euro per month) enables them to hold their all-hands Monday video meetings. They love the new notifications binary too because it makes the Nextcloud Talk notifications "instantaneous"... especially, when receiving a call, etc.

With these two HPB's enabled, Nextcloud is arguably the best, open source, self-hostable, remote-work collaboration tool available today. Without them....well, it great for managing files....like Owncloud, Pydio, etc.

If Cloudron devs intend their Nextcloud installation to be used for collaborative, remote work, then the notifications HPB binary needs to be enabled by default. It requires a decent amount of of server-side jugglery.

[avatar1024]

@jdaviescoates I'm not actually sure what it involves to enable this but do we know if anyone is working on it? Is it actually a separate app or just some specific config with a reverse proxy and redis? Does anyone know what are the missing pieces for it to work on Cloudron?

[oj]

@avatar1024 This docker-compose example on Reddit indicates the need for an additional container + an open port + env variables.

https://www.reddit.com/r/NextCloud/comments/m3emm9/nextcloud_with_hpb_dockercompose_example/

[jdaviescoates]

@avatar1024 I think @doodlemania2 may be offering this as a service (although perhaps that's just the High Performance Talk Backend, can't remember)

[necrevistonnezr]

@jdaviescoates It's the talk backend not this: https://forum.cloudron.io/post/30474

[doodlemania2]

@necrevistonnezr / @jdaviescoates - yeah, I do offer Talk - about 5 customers using it between US and EU servers now without any issue.

I tried to look at doing the files high performance one too. I need to revisit it - when I initially looked at it, I was like, yeah, no lol, but can't remember why. Will research!

[avatar1024]

@oj Great thanks.
@staff @appdev Do you know if all those requirements can be baked in to the current Cloudron NC app or would it work best as an additional app that links with NC?

[oj]

"Syncing at a new level: 10x to 100x": With Instant Sync thanks to the High Performance Back-end for Files introduced earlier this year, and the File Packing coming to V2 of the Sync Engine, both user experience and server load will dramatically improve.

For the server, idle load will have decreased by 10x thanks to the HPB, while during syncing the number of connections is also reduced by about a factor 10!

Users will benefit from instant availability of their files, even if they need to have large numbers of small files made available locally.

Nextcloud Sync v2 will be made available as part of a desktop client release later this year, alongside the release of Nextcloud 23.

[avatar1024]

With nextcloud 23 it seems there is again more improvement on performance for files, but it probably still requires the high performance back-end.

Has anyone looked into how to enable this for NC Cloudron package?

It feels like if we don't have it we're going to fall behind other NC instances which would be a shame.

[nebulon]

@avatar1024 from my perspective, stability is more important than speed when it comes to Nextcloud features and updates. So unless their high performance backend becomes the default, I would rather not set this for our app package with the experience we have had in the past with Nextcloud.

[avatar1024]

@nebulon Sure, I agree stability is key, and we should definitely take our time with NC update!

However, HPB for files has been there since NC21 and I wasn't aware of stability issues with it. You're right that the app (notify push) is not installed by default but I think that's only because it requires extra config on the server. So I don't think we need to modify the Cloudron NC package as such but "only" create an easy way to make the appropriate change on the server. I might be wrong though.

Procedure feels fairly simple on a stand alone NC instance but I don't know how that would apply to a cloudron set-up:

(relevant part at 32:39)

[jdaviescoates]

@nebulon said in High Performance Back-end for Nextcloud Files:

stability is more important than speed

Yes, but slow speed (especially of the web ui) is also the worst thing about Nextcloud.

So anything that speeds it up would be great!

Stability is more important overall, but speed is also very important imho.

IMHO the High Performance Backend should probably be added to the Cloudron Nextcloud package (or otherwise made easy to set-up on Cloudron), but probably not just yet.

Perhaps we should wait until notify_push (as it's called on Github) is at 1.0.1? (it's currently on 0.3.0) and/ or until Client Push (as it's called in Nextcloud Apps) has a positive overall rating? (currently neutral, see https://apps.nextcloud.com/apps/notify_push ).

[jdaviescoates]

@jdaviescoates said in High Performance Back-end for Nextcloud Files:

Perhaps we should wait until notify_push (as it's called on Github) is at 1.0.1? (it's currently on 0.3.0) and/ or until Client Push (as it's called in Nextcloud Apps) has a positive overall rating? (currently neutral, see https://apps.nextcloud.com/apps/notify_push ).

But, in the meantime it'd still be nice to be able to simply be able enable it on Cloudron too.

@avatar1024 have you tried getting it to work by following those instructions?

[avatar1024]

@jdaviescoates said in High Performance Back-end for Nextcloud Files:

Perhaps we should wait until notify_push (as it's called on Github) is at 1.0.1? (it's currently on 0.3.0) and/ or until Client Push (as it's called in Nextcloud Apps) has a positive overall rating? (currently neutral, see https://apps.nextcloud.com/apps/notify_push ).

Yes but this is only the NC app side of things. People can choose to enable it or not like any other app if there is any issue with it (plenty of apps are below the 1.0.0 version). So the question is, do the changes required on the server config have an impact if the app is disabled? It didn't look like that it would be the case.

@jdaviescoates said in High Performance Back-end for Nextcloud Files:

Yes, but slow speed (especially of the web ui) is also the worst thing about Nextcloud.

Agreed

@jdaviescoates said in High Performance Back-end for Nextcloud Files:

@avatar1024 have you tried getting it to work by following those instructions?

I didn't because this tuto is for a native install (not in docker) and so I don;t know how some of these translate in the case of a docker / containerised install like the cloudron one.

If the process is easily adaptable, then activating the HBP for files could just be as simple as adding the steps on how to do it in the Cloudron NC package documentation. And there again, users can choose to go with it or not. It might not require any change to the app package itself.

[jdaviescoates]

@avatar1024 said in High Performance Back-end for Nextcloud Files:

People can choose to enable it or not like any other app if there is any issue with it (plenty of apps are below the 1.0.0 version).

Fair point.

[avatar1024]

@jdaviescoates I have now tried to just follow the basic instruction instruction on the notify push github.

What I have done:

1. Installed the app inside NC
2. run occ notify_push:setup inside the app terminal
3. The abovce generated a systemd service like

[Unit]
Description = Push daemon for Nextcloud clients

[Service]
Environment=PORT=7867
Environment=NEXTCLOUD_URL=https://files.xxxxx.net/
ExecStart=/app/data/apps/notify_push/bin/x86_64/notify_push /app/code/config/config.php
User=www-data

[Install]
WantedBy = multi-user.target

I have replaced the path to the actual path to the binary and the config file like so

[Unit]
Description = Push daemon for Nextcloud clients

[Service]
Environment=PORT=7867
Environment=NEXTCLOUD_URL=https://files.xxxxx.net/
ExecStart=/home/yellowtent/appsdata/1445e2c5-fa60-4ce9-894e-3e2a8b8dd0a0/data/apps/notify_push/bin/x86_64/notify_push /home/yellowtent/appsdata/1445e2c5-fa60-4ce9-894e-3e2a8b8dd0a0/data/config/config.php
User=www-data

[Install]
WantedBy = multi-user.target

ran sudo systemctl enable --now notify_push as indicated.

But then it says the binary is not running.... so for now it's a fail. Anyone got any idea?

[girish]

@avatar1024 said in High Performance Back-end for Nextcloud Files:

But then it says the binary is not running

Maybe systemctl status notify_push ? and journalctl -u notify_push has more info.

[avatar1024]

@girish

systemctl status notify_push gives me:

notify_push.service - Push daemon for Nextcloud clients
     Loaded: loaded (/etc/systemd/system/notify_push.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Fri 2022-04-01 09:58:44 UTC; 14h ago
    Process: 180019 ExecStart=/home/yellowtent/appsdata/1445e2c5-fa60-4ce9-894e-3e2a8b8dd0a0/data/apps/notify_push/bin/x86_64/notify_push /home/yellowtent/appsdata/1445e2c5-fa60-4ce9-894e-3e2a8b8dd0a0/data/config/config.php (code=exited, status=1/FAILURE)
   Main PID: 180019 (code=exited, status=1/FAILURE)

Apr 01 09:58:44 v2202201132182176931 notify_push[180019]:    0: Failed to connect to Nextcloud database
Apr 01 09:58:44 v2202201132182176931 notify_push[180019]:    1: error communicating with the server: failed to lookup address information: Name does not resolve
Apr 01 09:58:44 v2202201132182176931 notify_push[180019]:    2: failed to lookup address information: Name does not resolve
Apr 01 09:58:44 v2202201132182176931 notify_push[180019]: Location:
Apr 01 09:58:44 v2202201132182176931 notify_push[180019]:    /tmp/krankerl.UyibIgt7EFoA/notify_push/src/storage_mapping.rs:58
Apr 01 09:58:44 v2202201132182176931 notify_push[180019]: Backtrace omitted.
Apr 01 09:58:44 v2202201132182176931 notify_push[180019]: Run with RUST_BACKTRACE=1 environment variable to display it.
Apr 01 09:58:44 v2202201132182176931 notify_push[180019]: Run with RUST_BACKTRACE=full to include source snippets.
Apr 01 09:58:44 v2202201132182176931 systemd[1]: notify_push.service: Main process exited, code=exited, status=1/FAILURE
Apr 01 09:58:44 v2202201132182176931 systemd[1]: notify_push.service: Failed with result 'exit-code'.

And journalctl -u notify_push

-- Logs begin at Mon 2022-03-28 21:45:49 UTC, end at Sat 2022-04-02 00:34:23 UTC. --
Apr 01 09:33:43 v2202201132182176931 systemd[1]: Started Push daemon for Nextcloud clients.
Apr 01 09:33:43 v2202201132182176931 notify_push[176600]: Error:
Apr 01 09:33:43 v2202201132182176931 notify_push[176600]:    0: Failed to parse config
Apr 01 09:33:43 v2202201132182176931 notify_push[176600]:    1: Failed to parse nextcloud config
Apr 01 09:33:43 v2202201132182176931 notify_push[176600]:    2: Failed to read config file
Apr 01 09:33:43 v2202201132182176931 notify_push[176600]: Location:
Apr 01 09:33:43 v2202201132182176931 notify_push[176600]:    src/config/nc.rs:8
Apr 01 09:33:43 v2202201132182176931 notify_push[176600]: Backtrace omitted.
Apr 01 09:33:43 v2202201132182176931 notify_push[176600]: Run with RUST_BACKTRACE=1 environment variable to display it.
Apr 01 09:33:43 v2202201132182176931 notify_push[176600]: Run with RUST_BACKTRACE=full to include source snippets.
Apr 01 09:33:43 v2202201132182176931 systemd[1]: notify_push.service: Main process exited, code=exited, status=1/FAILURE
Apr 01 09:33:43 v2202201132182176931 systemd[1]: notify_push.service: Failed with result 'exit-code'.
Apr 01 09:34:44 v2202201132182176931 systemd[1]: Started Push daemon for Nextcloud clients.
Apr 01 09:34:44 v2202201132182176931 notify_push[176680]: Error:
Apr 01 09:34:44 v2202201132182176931 notify_push[176680]:    0: Failed to parse config
Apr 01 09:34:44 v2202201132182176931 notify_push[176680]:    1: Failed to parse nextcloud config
Apr 01 09:34:44 v2202201132182176931 notify_push[176680]:    2: Failed to read config file
Apr 01 09:34:44 v2202201132182176931 notify_push[176680]: Location:
Apr 01 09:34:44 v2202201132182176931 notify_push[176680]:    src/config/nc.rs:8
Apr 01 09:34:44 v2202201132182176931 notify_push[176680]: Backtrace omitted.
Apr 01 09:34:44 v2202201132182176931 notify_push[176680]: Run with RUST_BACKTRACE=1 environment variable to display it.
Apr 01 09:34:44 v2202201132182176931 notify_push[176680]: Run with RUST_BACKTRACE=full to include source snippets.
Apr 01 09:34:44 v2202201132182176931 systemd[1]: notify_push.service: Main process exited, code=exited, status=1/FAILURE
Apr 01 09:34:44 v2202201132182176931 systemd[1]: notify_push.service: Failed with result 'exit-code'.
Apr 01 09:35:05 v2202201132182176931 notify_push[176786]: Location:
Apr 01 09:35:05 v2202201132182176931 notify_push[176786]:    src/config/nc.rs:8
Apr 01 09:35:05 v2202201132182176931 notify_push[176786]: Backtrace omitted.
Apr 01 09:35:05 v2202201132182176931 notify_push[176786]: Run with RUST_BACKTRACE=1 environment variable to display it.
Apr 01 09:35:05 v2202201132182176931 notify_push[176786]: Run with RUST_BACKTRACE=full to include source snippets.
Apr 01 09:35:05 v2202201132182176931 systemd[1]: notify_push.service: Main process exited, code=exited, status=1/FAILURE
Apr 01 09:35:05 v2202201132182176931 systemd[1]: notify_push.service: Failed with result 'exit-code'.
Apr 01 09:54:28 v2202201132182176931 systemd[1]: Started Push daemon for Nextcloud clients.
Apr 01 09:54:28 v2202201132182176931 notify_push[179480]: Error:
Apr 01 09:54:28 v2202201132182176931 notify_push[179480]:    0: Failed to connect to Nextcloud database
Apr 01 09:54:28 v2202201132182176931 notify_push[179480]:    1: error communicating with the server: failed to lookup address information: Name does not resolve
Apr 01 09:54:28 v2202201132182176931 notify_push[179480]:    2: failed to lookup address information: Name does not resolve
Apr 01 09:54:28 v2202201132182176931 notify_push[179480]: Location:
Apr 01 09:54:28 v2202201132182176931 notify_push[179480]:    /tmp/krankerl.UyibIgt7EFoA/notify_push/src/storage_mapping.rs:58
Apr 01 09:54:28 v2202201132182176931 notify_push[179480]: Backtrace omitted.
Apr 01 09:54:28 v2202201132182176931 notify_push[179480]: Run with RUST_BACKTRACE=1 environment variable to display it.
Apr 01 09:54:28 v2202201132182176931 notify_push[179480]: Run with RUST_BACKTRACE=full to include source snippets.
Apr 01 09:54:28 v2202201132182176931 systemd[1]: notify_push.service: Main process exited, code=exited, status=1/FAILURE
Apr 01 09:54:28 v2202201132182176931 systemd[1]: notify_push.service: Failed with result 'exit-code'.
Apr 01 09:58:44 v2202201132182176931 systemd[1]: Started Push daemon for Nextcloud clients.
Apr 01 09:58:44 v2202201132182176931 notify_push[180019]: Error:
Apr 01 09:58:44 v2202201132182176931 notify_push[180019]:    0: Failed to connect to Nextcloud database
Apr 01 09:58:44 v2202201132182176931 notify_push[180019]:    1: error communicating with the server: failed to lookup address information: Name does not resolve
Apr 01 09:58:44 v2202201132182176931 notify_push[180019]:    2: failed to lookup address information: Name does not resolve
Apr 01 09:58:44 v2202201132182176931 notify_push[180019]: Location:
Apr 01 09:58:44 v2202201132182176931 notify_push[180019]:    /tmp/krankerl.UyibIgt7EFoA/notify_push/src/storage_mapping.rs:58
Apr 01 09:58:44 v2202201132182176931 notify_push[180019]: Backtrace omitted.
Apr 01 09:58:44 v2202201132182176931 notify_push[180019]: Run with RUST_BACKTRACE=1 environment variable to display it.
Apr 01 09:58:44 v2202201132182176931 notify_push[180019]: Run with RUST_BACKTRACE=full to include source snippets.
Apr 01 09:58:44 v2202201132182176931 systemd[1]: notify_push.service: Main process exited, code=exited, status=1/FAILURE
Apr 01 09:58:44 v2202201132182176931 systemd[1]: notify_push.service: Failed with result 'exit-code'.

[girish]

@avatar1024 said in High Performance Back-end for Nextcloud Files:

/home/yellowtent/appsdata/1445e2c5-fa60-4ce9-894e-3e2a8b8dd0a0/data/apps/notify_push/bin/x86_64/notify_push /home/yellowtent/appsdata/1445e2c5-fa60-4ce9-894e-3e2a8b8dd0a0/data/config/config.php

You should try running that command standalone first. Looks like that doesn't work. (It's unable to locate configs or something?)

[eganonoa]

@girish @avatar1024 I was trying to follow this just now and am confused how you even got that far. It seems like @avatar1024 was able to create the systemd unit file, but I can't get anywhere with that because it's a readonly file system. Am I missing something here? Is there a way to place a new systemd unit file within a cloudron docker container? [Otherwise, I'm wondering whether @avatar1024 put the unit file into the base of the cloudron server, and not the docker container, which would explain why it doesn't find the relevant binary and config].

It would be really useful to figure this out and understand where and how I must put a systemd unit file within a read-only container. Or if not that to otherwise know how to create a binary that won't get overwritten on restart, but would rather autostart (see e.g. how linuxserver containers do this: https://github.com/linuxserver/docker-nextcloud/issues/194)

Beyond that, it looks like I have been able to get notify push to work by simply calling the binary itself within the nextcloud container. The steps I took were as follows (following this guide https://github.com/nextcloud/notify_push) :

1. Install and enable notify_push app in Nextcloud
2. In the terminal of the nextcloud container directly run the binary with the relevant variables (./app/data/apps/notify_push/bin/x86_64/notify_push --allow-self-signed --nextcloud-url https://[Nextcloud URL] --port 7867 -- /app/data/config/config.php)
3. In the overall cloudron server edit the relevant nginx config for the Nextcloud container to add the location block as per the github readme (changing the IP address to that of the container) and reloading nginx
4. Open another terminal for the nextcloud container and add the relevant IP address to the trusted proxies array in the Nextcloud config.php
5. Enable the app (occ app:enable notify_push)
6. set the url of the push server (occ notify_push:setup https://url/push)

Once done I get a nice confirmation that everything is working with green ticks for all of the relevant parts:

✓ redis is configured
✓ push server is receiving redis messages
✓ push server can load mount info from database
✓ push server can connect to the Nextcloud server
✓ push server is a trusted proxy
✓ push server is running the same version as the app

And when I run occ notify_push:metrics it gives me metrics that appear to confirm that everything is working well.

So, as I say, it all seems quite possible, though nothing is going to survive a reboot and I'd love to find a way to at least autostart running the binary if not setting it all up as a systemd service within the docker container. Any help would be much appreciated.