Integrating Syncthing with a web-base file manager / Nextcloud files / Pydio

[avatar1024]

Hello,

When it comes to syncing files, Synthing outperform Nextcloud by a long stretch. However with syncthing installed on the server you cannot access your files via a web interface, as you would do with Nextcloud.

Two main approached to integrate the tow are:

1. to manually give permission to syncthing on your nextcloud folder on your server, and add that folder as a sync in thinking. Issues with this is that the files synced with Syncthing won't be visible in the nextcloud interface unless you trigger manually a rescan of that folder with a nextcloud command (occ files:scan --all). This process is resource intensive as it rescans all files rather than the one that changes. If you use the nextcloud web interface a lot you'd have to schedule that command to run regularly. So overall quite inefficient. Also it some limitations in terms of integrate file versioning and trash between the two apps. Now someone clever wrote a script to file watch modification made by syncthing, and address the versioning issue. See here and here. Issue is it hasn't been maintained for 5 years, though was still working fine 3 years ago. Might be a good place to start though as I reckon things might not have changed much in the architecture of both apps on that front.
   Apparently there would not be such a problem using Pydio instead of Nextcloud. Not yet on Cloudron though.

2. Adding Syncthing folder as external storage. Apparently works fine in terms of detecting changes right away but has some limitations again in terms file versioning and trash. Not the end of the world in many use case (you just have two file versioning systems and two trash systems). THE issue here is that external storage is not support in the Cloudron version of Nextcloud

Relevant post on the topic if people wanna look into this. I think it'd be great to come up with a solution for users of Nextcloud on Cloudron. I;d be up to help testing stuff. Sadly I cannot code...

https://forum.syncthing.net/t/integrate-syncthing-with-nextcloud-and-owncloud/11289
https://forum.syncthing.net/t/integrating-syncthing-with-owncloud/6332
https://forum.syncthing.net/t/mixed-syncthing-with-nextcloud/10499/3
https://github.com/nextcloud/server/issues/8384
https://help.nextcloud.com/t/syncthing-nextcloud-how-to/5459

[jdaviescoates]

@avatar1024 said in Integrating Syncthing with a web-base file manager / Nextcloud files / Pydio:

THE issue here is that external storage is not support in the Cloudron version of Nextcloud

Hmm, yeah:

@girish said in What's coming in 4.5:

Or do you want to mount an external disk into nextcloud with the external storage app ? That is indeed not possible today.

Although I wonder if we could make use of Custom Data Directory for this use case?

https://cloudron.io/documentation/storage/#app-data-directory

[will]

This sounds like a absolute nightmare. NextCloud external storage has been half baked at best in my experience, tying in syncthing sounds like a massive PITA.
Were you able to get this working? And SyncThing doesn't necessarily out perform Next Cloud. It depends on what you're doing with the data, and how large it is, and block sizes and stuff.

[avatar1024]

@will Haven't tried it myself as I only use NC in cloudron and so it's not possible to have external storage. But people reporting on forums (some of the link provided above) said it was straightforward, just mount your syncthing folder as external storage and it just works, apparently.

Do you have a use cases (or file type) where NC outperform syncthing? I'd like to try and understand this to optimise my setup as in my experience (I used to have basically all of my home directory synced with synthing so with various file size, from documents to pictures, to music and videos), syncthing has been much faster than Nextcloud at syncing between devices. I mostly used Nextcloud for documents though, some of which are large but not above a 500mb for any single file.

[will]

@avatar1024 A lot of the same thought process as here, on block vs file sync.

https://www.enterprisestorageforum.com/storage-networking/nas-vs.-san-differences-and-use-cases.html

"SANs are the higher performers for environments that need high-speed traffic such as high transaction databases and ecommerce websites. NAS generally has lower throughput and higher latency because of its slower file system layer, but high-speed networks can make up for performance losses within NAS."

Here is the thing, Nextcloud is half baked on just about everything outside of file sync. I'd be VERY wary to trust it to handle external syncing. It might be easy to setup, but you're betting on their external storage plugin working as advertised under all conditions.

An interesting idea to be sure!