Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.


  • Categories
  • Recent
  • Tags
  • Popular
  • Bookmarks
Skins
  • Light
  • Cerulean
  • Cosmo
  • Flatly
  • Journal
  • Litera
  • Lumen
  • Lux
  • Materia
  • Minty
  • Morph
  • Pulse
  • Sandstone
  • Simplex
  • Sketchy
  • Spacelab
  • United
  • Yeti
  • Zephyr
  • Dark
  • Cyborg
  • Darkly
  • Quartz
  • Slate
  • Solar
  • Superhero
  • Vapor

  • Default (No Skin)
  • No Skin
Collapse

Cloudron Forum

Apps | Demo | Docs | Install

High Performance Back-end for Nextcloud Files

Scheduled Pinned Locked Moved App Wishlist
43 Posts 9 Posters 2.5k Views
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • jdaviescoatesJ Offline
    jdaviescoatesJ Offline
    jdaviescoates
    wrote on last edited by
    #1

    There is already an App Wishlist post for the Nextcloud Talk High Performance Back-end but I figured I'd add one for the High Performance Back-end for Nextcloud Files too.

    https://nextcloud.com/blog/nextcloud-faster-than-ever-introducing-files-high-performance-back-end/
    https://github.com/nextcloud/notify_push

    It'd be great to have this on Cloudron!

    As @oj said in Nextcloud 21 is out:

    @girish @nebulon For users who are using Nextcloud + Nextcloud Talk for remote, collaborative work, the two Nextcloud HPB's (the Spreed SFU/ signalling HPB + this new notifications HPB binary) are very important.

    My largest (non-Cloudron) Nextcloud has 22 remote users; the Spreed SFU (available as-a-service for 40 euro per month) enables them to hold their all-hands Monday video meetings. They love the new notifications binary too because it makes the Nextcloud Talk notifications "instantaneous"... especially, when receiving a call, etc.

    With these two HPB's enabled, Nextcloud is arguably the best, open source, self-hostable, remote-work collaboration tool available today. Without them....well, it great for managing files....like Owncloud, Pydio, etc.

    If Cloudron devs intend their Nextcloud installation to be used for collaborative, remote work, then the notifications HPB binary needs to be enabled by default. It requires a decent amount of of server-side jugglery.

    I use Cloudron with Gandi & Hetzner

    avatar1024A O 2 Replies Last reply
    5
  • avatar1024A Offline
    avatar1024A Offline
    avatar1024
    replied to jdaviescoates on last edited by
    #2

    @jdaviescoates I'm not actually sure what it involves to enable this but do we know if anyone is working on it? Is it actually a separate app or just some specific config with a reverse proxy and redis? Does anyone know what are the missing pieces for it to work on Cloudron?

    O jdaviescoatesJ 2 Replies Last reply
    2
  • O Offline
    O Offline
    oj
    replied to avatar1024 on last edited by
    #3

    @avatar1024 This docker-compose example on Reddit indicates the need for an additional container + an open port + env variables.

    https://www.reddit.com/r/NextCloud/comments/m3emm9/nextcloud_with_hpb_dockercompose_example/

    avatar1024A 1 Reply Last reply
    2
  • jdaviescoatesJ Offline
    jdaviescoatesJ Offline
    jdaviescoates
    replied to avatar1024 on last edited by
    #4

    @avatar1024 I think @doodlemania2 may be offering this as a service (although perhaps that's just the High Performance Talk Backend, can't remember)

    I use Cloudron with Gandi & Hetzner

    necrevistonnezrN 1 Reply Last reply
    1
  • necrevistonnezrN Offline
    necrevistonnezrN Offline
    necrevistonnezr
    replied to jdaviescoates on last edited by
    #5

    @jdaviescoates It's the talk backend not this: https://forum.cloudron.io/post/30474

    doodlemania2D 1 Reply Last reply
    1
  • doodlemania2D Offline
    doodlemania2D Offline
    doodlemania2 App Dev
    replied to necrevistonnezr on last edited by
    #6

    @necrevistonnezr / @jdaviescoates - yeah, I do offer Talk - about 5 customers using it between US and EU servers now without any issue.

    I tried to look at doing the files high performance one too. I need to revisit it - when I initially looked at it, I was like, yeah, no lol, but can't remember why. Will research!

    1 Reply Last reply
    3
  • avatar1024A Offline
    avatar1024A Offline
    avatar1024
    replied to oj on last edited by
    #7

    @oj Great thanks.
    @staff @appdev Do you know if all those requirements can be baked in to the current Cloudron NC app or would it work best as an additional app that links with NC?

    1 Reply Last reply
    1
  • O Offline
    O Offline
    oj
    replied to jdaviescoates on last edited by
    #8

    "Syncing at a new level: 10x to 100x": With Instant Sync thanks to the High Performance Back-end for Files introduced earlier this year, and the File Packing coming to V2 of the Sync Engine, both user experience and server load will dramatically improve.

    For the server, idle load will have decreased by 10x thanks to the HPB, while during syncing the number of connections is also reduced by about a factor 10!

    Users will benefit from instant availability of their files, even if they need to have large numbers of small files made available locally.

    Nextcloud Sync v2 will be made available as part of a desktop client release later this year, alongside the release of Nextcloud 23.

    1 Reply Last reply
    2
  • avatar1024A Offline
    avatar1024A Offline
    avatar1024
    wrote on last edited by
    #9

    With nextcloud 23 it seems there is again more improvement on performance for files, but it probably still requires the high performance back-end.

    Has anyone looked into how to enable this for NC Cloudron package?

    It feels like if we don't have it we're going to fall behind other NC instances which would be a shame.

    nebulonN 1 Reply Last reply
    1
  • nebulonN Online
    nebulonN Online
    nebulon Staff
    replied to avatar1024 on last edited by
    #10

    @avatar1024 from my perspective, stability is more important than speed when it comes to Nextcloud features and updates. So unless their high performance backend becomes the default, I would rather not set this for our app package with the experience we have had in the past with Nextcloud.

    avatar1024A jdaviescoatesJ 2 Replies Last reply
    3
  • avatar1024A Offline
    avatar1024A Offline
    avatar1024
    replied to nebulon on last edited by avatar1024
    #11

    @nebulon Sure, I agree stability is key, and we should definitely take our time with NC update!

    However, HPB for files has been there since NC21 and I wasn't aware of stability issues with it. You're right that the app (notify push) is not installed by default but I think that's only because it requires extra config on the server. So I don't think we need to modify the Cloudron NC package as such but "only" create an easy way to make the appropriate change on the server. I might be wrong though.

    Procedure feels fairly simple on a stand alone NC instance but I don't know how that would apply to a cloudron set-up:

    (relevant part at 32:39)

    1 Reply Last reply
    1
  • jdaviescoatesJ Offline
    jdaviescoatesJ Offline
    jdaviescoates
    replied to nebulon on last edited by
    #12

    @nebulon said in High Performance Back-end for Nextcloud Files:

    stability is more important than speed

    Yes, but slow speed (especially of the web ui) is also the worst thing about Nextcloud.

    So anything that speeds it up would be great!

    Stability is more important overall, but speed is also very important imho.

    IMHO the High Performance Backend should probably be added to the Cloudron Nextcloud package (or otherwise made easy to set-up on Cloudron), but probably not just yet.

    Perhaps we should wait until notify_push (as it's called on Github) is at 1.0.1? (it's currently on 0.3.0) and/ or until Client Push (as it's called in Nextcloud Apps) has a positive overall rating? (currently neutral, see https://apps.nextcloud.com/apps/notify_push ).

    I use Cloudron with Gandi & Hetzner

    jdaviescoatesJ avatar1024A 2 Replies Last reply
    2
  • jdaviescoatesJ Offline
    jdaviescoatesJ Offline
    jdaviescoates
    replied to jdaviescoates on last edited by jdaviescoates
    #13

    @jdaviescoates said in High Performance Back-end for Nextcloud Files:

    Perhaps we should wait until notify_push (as it's called on Github) is at 1.0.1? (it's currently on 0.3.0) and/ or until Client Push (as it's called in Nextcloud Apps) has a positive overall rating? (currently neutral, see https://apps.nextcloud.com/apps/notify_push ).

    But, in the meantime it'd still be nice to be able to simply be able enable it on Cloudron too.

    @avatar1024 have you tried getting it to work by following those instructions?

    I use Cloudron with Gandi & Hetzner

    avatar1024A 1 Reply Last reply
    1
  • avatar1024A Offline
    avatar1024A Offline
    avatar1024
    replied to jdaviescoates on last edited by avatar1024
    #14

    @jdaviescoates said in High Performance Back-end for Nextcloud Files:

    Perhaps we should wait until notify_push (as it's called on Github) is at 1.0.1? (it's currently on 0.3.0) and/ or until Client Push (as it's called in Nextcloud Apps) has a positive overall rating? (currently neutral, see https://apps.nextcloud.com/apps/notify_push ).

    Yes but this is only the NC app side of things. People can choose to enable it or not like any other app if there is any issue with it (plenty of apps are below the 1.0.0 version). So the question is, do the changes required on the server config have an impact if the app is disabled? It didn't look like that it would be the case.

    @jdaviescoates said in High Performance Back-end for Nextcloud Files:

    Yes, but slow speed (especially of the web ui) is also the worst thing about Nextcloud.

    Agreed 😁

    @jdaviescoates said in High Performance Back-end for Nextcloud Files:

    @avatar1024 have you tried getting it to work by following those instructions?

    I didn't because this tuto is for a native install (not in docker) and so I don;t know how some of these translate in the case of a docker / containerised install like the cloudron one.

    If the process is easily adaptable, then activating the HBP for files could just be as simple as adding the steps on how to do it in the Cloudron NC package documentation. And there again, users can choose to go with it or not. It might not require any change to the app package itself.

    jdaviescoatesJ 1 Reply Last reply
    1
  • jdaviescoatesJ Offline
    jdaviescoatesJ Offline
    jdaviescoates
    replied to avatar1024 on last edited by
    #15

    @avatar1024 said in High Performance Back-end for Nextcloud Files:

    People can choose to enable it or not like any other app if there is any issue with it (plenty of apps are below the 1.0.0 version).

    Fair point.

    I use Cloudron with Gandi & Hetzner

    1 Reply Last reply
    0
  • avatar1024A Offline
    avatar1024A Offline
    avatar1024
    replied to jdaviescoates on last edited by
    #16

    @jdaviescoates I have now tried to just follow the basic instruction instruction on the notify push github.

    What I have done:

    1. Installed the app inside NC
    2. run occ notify_push:setup inside the app terminal
    3. The abovce generated a systemd service like
    [Unit]
    Description = Push daemon for Nextcloud clients
    
    [Service]
    Environment=PORT=7867
    Environment=NEXTCLOUD_URL=https://files.xxxxx.net/
    ExecStart=/app/data/apps/notify_push/bin/x86_64/notify_push /app/code/config/config.php
    User=www-data
    
    [Install]
    WantedBy = multi-user.target
    

    I have replaced the path to the actual path to the binary and the config file like so

    [Unit]
    Description = Push daemon for Nextcloud clients
    
    [Service]
    Environment=PORT=7867
    Environment=NEXTCLOUD_URL=https://files.xxxxx.net/
    ExecStart=/home/yellowtent/appsdata/1445e2c5-fa60-4ce9-894e-3e2a8b8dd0a0/data/apps/notify_push/bin/x86_64/notify_push /home/yellowtent/appsdata/1445e2c5-fa60-4ce9-894e-3e2a8b8dd0a0/data/config/config.php
    User=www-data
    
    [Install]
    WantedBy = multi-user.target
    
    

    ran sudo systemctl enable --now notify_push as indicated.

    But then it says the binary is not running.... so for now it's a fail. Anyone got any idea?

    girishG 1 Reply Last reply
    3
  • girishG Do not disturb
    girishG Do not disturb
    girish Staff
    replied to avatar1024 on last edited by
    #17

    @avatar1024 said in High Performance Back-end for Nextcloud Files:

    But then it says the binary is not running

    Maybe systemctl status notify_push ? and journalctl -u notify_push has more info.

    avatar1024A 1 Reply Last reply
    1
  • avatar1024A Offline
    avatar1024A Offline
    avatar1024
    replied to girish on last edited by
    #18

    @girish

    systemctl status notify_push gives me:

    notify_push.service - Push daemon for Nextcloud clients
         Loaded: loaded (/etc/systemd/system/notify_push.service; enabled; vendor preset: enabled)
         Active: failed (Result: exit-code) since Fri 2022-04-01 09:58:44 UTC; 14h ago
        Process: 180019 ExecStart=/home/yellowtent/appsdata/1445e2c5-fa60-4ce9-894e-3e2a8b8dd0a0/data/apps/notify_push/bin/x86_64/notify_push /home/yellowtent/appsdata/1445e2c5-fa60-4ce9-894e-3e2a8b8dd0a0/data/config/config.php (code=exited, status=1/FAILURE)
       Main PID: 180019 (code=exited, status=1/FAILURE)
    
    Apr 01 09:58:44 v2202201132182176931 notify_push[180019]:    0: Failed to connect to Nextcloud database
    Apr 01 09:58:44 v2202201132182176931 notify_push[180019]:    1: error communicating with the server: failed to lookup address information: Name does not resolve
    Apr 01 09:58:44 v2202201132182176931 notify_push[180019]:    2: failed to lookup address information: Name does not resolve
    Apr 01 09:58:44 v2202201132182176931 notify_push[180019]: Location:
    Apr 01 09:58:44 v2202201132182176931 notify_push[180019]:    /tmp/krankerl.UyibIgt7EFoA/notify_push/src/storage_mapping.rs:58
    Apr 01 09:58:44 v2202201132182176931 notify_push[180019]: Backtrace omitted.
    Apr 01 09:58:44 v2202201132182176931 notify_push[180019]: Run with RUST_BACKTRACE=1 environment variable to display it.
    Apr 01 09:58:44 v2202201132182176931 notify_push[180019]: Run with RUST_BACKTRACE=full to include source snippets.
    Apr 01 09:58:44 v2202201132182176931 systemd[1]: notify_push.service: Main process exited, code=exited, status=1/FAILURE
    Apr 01 09:58:44 v2202201132182176931 systemd[1]: notify_push.service: Failed with result 'exit-code'.
    
    

    And journalctl -u notify_push

    -- Logs begin at Mon 2022-03-28 21:45:49 UTC, end at Sat 2022-04-02 00:34:23 UTC. --
    Apr 01 09:33:43 v2202201132182176931 systemd[1]: Started Push daemon for Nextcloud clients.
    Apr 01 09:33:43 v2202201132182176931 notify_push[176600]: Error:
    Apr 01 09:33:43 v2202201132182176931 notify_push[176600]:    0: Failed to parse config
    Apr 01 09:33:43 v2202201132182176931 notify_push[176600]:    1: Failed to parse nextcloud config
    Apr 01 09:33:43 v2202201132182176931 notify_push[176600]:    2: Failed to read config file
    Apr 01 09:33:43 v2202201132182176931 notify_push[176600]: Location:
    Apr 01 09:33:43 v2202201132182176931 notify_push[176600]:    src/config/nc.rs:8
    Apr 01 09:33:43 v2202201132182176931 notify_push[176600]: Backtrace omitted.
    Apr 01 09:33:43 v2202201132182176931 notify_push[176600]: Run with RUST_BACKTRACE=1 environment variable to display it.
    Apr 01 09:33:43 v2202201132182176931 notify_push[176600]: Run with RUST_BACKTRACE=full to include source snippets.
    Apr 01 09:33:43 v2202201132182176931 systemd[1]: notify_push.service: Main process exited, code=exited, status=1/FAILURE
    Apr 01 09:33:43 v2202201132182176931 systemd[1]: notify_push.service: Failed with result 'exit-code'.
    Apr 01 09:34:44 v2202201132182176931 systemd[1]: Started Push daemon for Nextcloud clients.
    Apr 01 09:34:44 v2202201132182176931 notify_push[176680]: Error:
    Apr 01 09:34:44 v2202201132182176931 notify_push[176680]:    0: Failed to parse config
    Apr 01 09:34:44 v2202201132182176931 notify_push[176680]:    1: Failed to parse nextcloud config
    Apr 01 09:34:44 v2202201132182176931 notify_push[176680]:    2: Failed to read config file
    Apr 01 09:34:44 v2202201132182176931 notify_push[176680]: Location:
    Apr 01 09:34:44 v2202201132182176931 notify_push[176680]:    src/config/nc.rs:8
    Apr 01 09:34:44 v2202201132182176931 notify_push[176680]: Backtrace omitted.
    Apr 01 09:34:44 v2202201132182176931 notify_push[176680]: Run with RUST_BACKTRACE=1 environment variable to display it.
    Apr 01 09:34:44 v2202201132182176931 notify_push[176680]: Run with RUST_BACKTRACE=full to include source snippets.
    Apr 01 09:34:44 v2202201132182176931 systemd[1]: notify_push.service: Main process exited, code=exited, status=1/FAILURE
    Apr 01 09:34:44 v2202201132182176931 systemd[1]: notify_push.service: Failed with result 'exit-code'.
    Apr 01 09:35:05 v2202201132182176931 notify_push[176786]: Location:
    Apr 01 09:35:05 v2202201132182176931 notify_push[176786]:    src/config/nc.rs:8
    Apr 01 09:35:05 v2202201132182176931 notify_push[176786]: Backtrace omitted.
    Apr 01 09:35:05 v2202201132182176931 notify_push[176786]: Run with RUST_BACKTRACE=1 environment variable to display it.
    Apr 01 09:35:05 v2202201132182176931 notify_push[176786]: Run with RUST_BACKTRACE=full to include source snippets.
    Apr 01 09:35:05 v2202201132182176931 systemd[1]: notify_push.service: Main process exited, code=exited, status=1/FAILURE
    Apr 01 09:35:05 v2202201132182176931 systemd[1]: notify_push.service: Failed with result 'exit-code'.
    Apr 01 09:54:28 v2202201132182176931 systemd[1]: Started Push daemon for Nextcloud clients.
    Apr 01 09:54:28 v2202201132182176931 notify_push[179480]: Error:
    Apr 01 09:54:28 v2202201132182176931 notify_push[179480]:    0: Failed to connect to Nextcloud database
    Apr 01 09:54:28 v2202201132182176931 notify_push[179480]:    1: error communicating with the server: failed to lookup address information: Name does not resolve
    Apr 01 09:54:28 v2202201132182176931 notify_push[179480]:    2: failed to lookup address information: Name does not resolve
    Apr 01 09:54:28 v2202201132182176931 notify_push[179480]: Location:
    Apr 01 09:54:28 v2202201132182176931 notify_push[179480]:    /tmp/krankerl.UyibIgt7EFoA/notify_push/src/storage_mapping.rs:58
    Apr 01 09:54:28 v2202201132182176931 notify_push[179480]: Backtrace omitted.
    Apr 01 09:54:28 v2202201132182176931 notify_push[179480]: Run with RUST_BACKTRACE=1 environment variable to display it.
    Apr 01 09:54:28 v2202201132182176931 notify_push[179480]: Run with RUST_BACKTRACE=full to include source snippets.
    Apr 01 09:54:28 v2202201132182176931 systemd[1]: notify_push.service: Main process exited, code=exited, status=1/FAILURE
    Apr 01 09:54:28 v2202201132182176931 systemd[1]: notify_push.service: Failed with result 'exit-code'.
    Apr 01 09:58:44 v2202201132182176931 systemd[1]: Started Push daemon for Nextcloud clients.
    Apr 01 09:58:44 v2202201132182176931 notify_push[180019]: Error:
    Apr 01 09:58:44 v2202201132182176931 notify_push[180019]:    0: Failed to connect to Nextcloud database
    Apr 01 09:58:44 v2202201132182176931 notify_push[180019]:    1: error communicating with the server: failed to lookup address information: Name does not resolve
    Apr 01 09:58:44 v2202201132182176931 notify_push[180019]:    2: failed to lookup address information: Name does not resolve
    Apr 01 09:58:44 v2202201132182176931 notify_push[180019]: Location:
    Apr 01 09:58:44 v2202201132182176931 notify_push[180019]:    /tmp/krankerl.UyibIgt7EFoA/notify_push/src/storage_mapping.rs:58
    Apr 01 09:58:44 v2202201132182176931 notify_push[180019]: Backtrace omitted.
    Apr 01 09:58:44 v2202201132182176931 notify_push[180019]: Run with RUST_BACKTRACE=1 environment variable to display it.
    Apr 01 09:58:44 v2202201132182176931 notify_push[180019]: Run with RUST_BACKTRACE=full to include source snippets.
    Apr 01 09:58:44 v2202201132182176931 systemd[1]: notify_push.service: Main process exited, code=exited, status=1/FAILURE
    Apr 01 09:58:44 v2202201132182176931 systemd[1]: notify_push.service: Failed with result 'exit-code'.
    
    
    girishG 1 Reply Last reply
    0
  • girishG Do not disturb
    girishG Do not disturb
    girish Staff
    replied to avatar1024 on last edited by
    #19

    @avatar1024 said in High Performance Back-end for Nextcloud Files:

    /home/yellowtent/appsdata/1445e2c5-fa60-4ce9-894e-3e2a8b8dd0a0/data/apps/notify_push/bin/x86_64/notify_push /home/yellowtent/appsdata/1445e2c5-fa60-4ce9-894e-3e2a8b8dd0a0/data/config/config.php

    You should try running that command standalone first. Looks like that doesn't work. (It's unable to locate configs or something?)

    E 1 Reply Last reply
    0
  • E Offline
    E Offline
    eganonoa
    replied to girish on last edited by eganonoa
    #20

    @girish @avatar1024 I was trying to follow this just now and am confused how you even got that far. It seems like @avatar1024 was able to create the systemd unit file, but I can't get anywhere with that because it's a readonly file system. Am I missing something here? Is there a way to place a new systemd unit file within a cloudron docker container? [Otherwise, I'm wondering whether @avatar1024 put the unit file into the base of the cloudron server, and not the docker container, which would explain why it doesn't find the relevant binary and config].

    It would be really useful to figure this out and understand where and how I must put a systemd unit file within a read-only container. Or if not that to otherwise know how to create a binary that won't get overwritten on restart, but would rather autostart (see e.g. how linuxserver containers do this: https://github.com/linuxserver/docker-nextcloud/issues/194)

    Beyond that, it looks like I have been able to get notify push to work by simply calling the binary itself within the nextcloud container. The steps I took were as follows (following this guide https://github.com/nextcloud/notify_push) :

    1. Install and enable notify_push app in Nextcloud
    2. In the terminal of the nextcloud container directly run the binary with the relevant variables (./app/data/apps/notify_push/bin/x86_64/notify_push --allow-self-signed --nextcloud-url https://[Nextcloud URL] --port 7867 -- /app/data/config/config.php)
    3. In the overall cloudron server edit the relevant nginx config for the Nextcloud container to add the location block as per the github readme (changing the IP address to that of the container) and reloading nginx
    4. Open another terminal for the nextcloud container and add the relevant IP address to the trusted proxies array in the Nextcloud config.php
    5. Enable the app (occ app:enable notify_push)
    6. set the url of the push server (occ notify_push:setup https://url/push)

    Once done I get a nice confirmation that everything is working with green ticks for all of the relevant parts:

    āœ“ redis is configured
    āœ“ push server is receiving redis messages
    āœ“ push server can load mount info from database
    āœ“ push server can connect to the Nextcloud server
    āœ“ push server is a trusted proxy
    āœ“ push server is running the same version as the app

    And when I run occ notify_push:metrics it gives me metrics that appear to confirm that everything is working well.

    So, as I say, it all seems quite possible, though nothing is going to survive a reboot and I'd love to find a way to at least autostart running the binary if not setting it all up as a systemd service within the docker container. Any help would be much appreciated.

    girishG 2 Replies Last reply
    3

  • Login

  • Don't have an account? Register

  • Login or register to search.
  • First post
    Last post
0
  • Categories
  • Recent
  • Tags
  • Popular
  • Bookmarks
  • Login

  • Don't have an account? Register

  • Login or register to search.