Apps for file management/sharing/syncing

[timconsidine]

@Joseph I didn't think of SFTPGo in that way, but yes, it could loosely be

[crazybrad]

@timconsidine I was thinking much the same. Currently using Box.com (good, expensive, excellent mobile app, NO LINUX CLIENT) and was wondering about whether Seafile could be packaged for Cloudron. I think you answered that neatly already.

Because of Cloudron's architecture, I think the ability to back up metadata about files from a database (built-in) and using an S3-compatible back-end store for files would be an ideal solution.

For me, a strong mobile app is a necessity. There have been countless times when someone asked for a file, and with the Box iOS app, I was able to get it to the recipient instantly.

I will keep looking. Was hoping that Pydio could be the one, but sounds like it's not quite ready.

[timconsidine]

@crazybrad there is a Pydio mobile app for iOS
Not explored it much yet but seems ok

Android : would need to check.

[crazybrad]

@timconsidine So I had a few minutes and "tasked" my right-hand assistant with the job of evaluating options. Here was her assignment:

Research all open source apps that can be used for file storage similar to Box.com and Dropbox. App must:

• Use S3-compatible storage as backend for files (object) storage.
• Support infinite versions.
• Have good iOS and Android mobile apps for sharing, moving and copying files.
• Docker installation option.

In addition, for those meeting all the above criteria, analyze the tech stack and determine whether packaging the app for Cloudron would require small, medium or large amount of work.

Also assess the frequency of updates and whether that would become a large burden for Cloudron maintainers to keep up.

Here were the results:

Executive Summary

S3 Storage Analysis

Cloudron Packaging

She missed that NextCloud was already packaged, but no one is perfect:)

Bottom line: Pydio Cells looks very viable:)

@joseph What are your thoughts from Team Cloudron's perspective?

[timconsidine]

@crazybrad interesting

Maybe I should take another look at viability of packaging Seafile.

Just realised that I am testing Pydio based on flat file structure (files exist natively on storage) not S3.
Need to look into S3.

[crazybrad]

@tim What I like about S3 (and compatibles) is the "five 9s" reliability of object storage and unlimited expansion capability. I guess I grew tired of managing local storage...

Seafile ongoing maintenance effort seemed concerning (assuming it's realistic). Curious about what Cloudron team thinks about maintaining either.

Also, Pydio seemed to earn some kudos for a lightweight, high-performing tech stack. The analysis seemed to suggest it really leverages the S3 API - which is solid at this point.

[timconsidine]

Hmmm, my AI say my earlier comment that Seafile "is impossible to package for Cloudron" is wrong.
So I will take a fresh look at packaging Seafile.

S3 backend needs the PRO edition of Seafile, but I seem to recall they offered a free PRO licence for small deployments (not sure if they still do).

[crazybrad]

@timconsidine I remember reading about a PRO license for small business (up to 9 seats) for $100 per year (self-hosted). So not free technically, but essentially free given the value provided.

[timconsidine]

Partial update : despite my previous view that Seafile was not viable (currently eating humble pie), I am close to having it packaged as a custom app.
If successful, this would be a game changer.
The only things I don't like about Seafile are :

• slightly dated UI, but hey ho, you get used to it and it is compensated for by working well
• 'cloaked' file names on the underlying storage, but will work on a export/backup facility after the base package is working satisfactorily.

[timconsidine]

@crazybrad yes, I checked, they are still offering PRO for small deployments - still needs a licence but zero-cost for up to 3

[crazybrad]

@timconsidine You are impressive my friend. Can't believe you were able to package this so quickly. Just curious: where do you think the AI estimates are inflated? Perhaps Tim is 4x faster than others:)

And you are 100% correct about the <= 3 scenario. I missed that.

[timconsidine]

@crazybrad don't jump to conclusions - it's not finished yet !
May turn out to be in line with estimates, who knows.
Will plough on - core build done but a fair few wrinkles need resolving, and of course lots of testing.

[crazybrad]

@timconsidine Understood. But as I recall, AI was projecting a 40+ hour effort. Perhaps the last few tasks will take a lot of time. Been there!

[timconsidine]

Seafile is now packaged as a custom app.
https://forum.cloudron.io/post/118275

@crazybrad : it definitely took longer than your assistant estimated, but I didn't track how long.

[crazybrad]

@timconsidine Gut feel, how much more (as a percent)? Just trying to calibrate AI as an estimating tool.

[timconsidine]

@crazybrad I don't think more than 80.
Maybe 60?
So maybe 150% ??

With hindsight (or better foresight) it's always easier to do a better job, but some apps are well documented as to their install and configure process, others not so much.
Seafile is generally good documentation but like many apps, their attitude is "use our Docker image", rather than here's how to build.
So 60 hours estimate involved a lot of blundering about.
But done now, and I'm really happy to have Seafile available as an option (used it for 3+ years off-Cloudron).

About to write a companion app which will pull Seafile libraries into backup storage as native files on external storage.
Can never have enough backups !

[timconsidine]

SeaClone now published as a custom app : https://forum.cloudron.io/post/118327

[crazybrad]

@timconsidine AI "guessed" 41-57 hours. So reality was just above the upper estimate. Also, not sure if AI "guessed" a PRO version instead of Community versions (which is what was possible).

On the topic of automated backup, if you store your SeaFile objects on Amazon S3, it is possible with bucket policies to replicate objects to a cheaper storage tier. No need to extend the application. And then you can have "deep storage" files delete automatically after a certain period of time. I don't know if some of the S3-compatible services like iDrive E2 and BackBlaze offer this capability but there are some interesting possibilities.

Finally, read an article that gave me pause about backups. A small company lost millions in revenue because their cloud provider (who shall be anonymous) went down and their backups were in the same availability region (which was part of the outage). This is making me think that perhaps duplicate backups (whether S3 objects or VPS snapshots might be worth replicating in another availability zone.

[timconsidine]

@crazybrad interesting comments

The packaging work for PRO might not actually be much higher, the blocker is that Seafile require the end-user to register for access to it, if I or Cloudron register, then share that file/link out, it's a breach of Seafile licence. So we have to rely on packaging CE and documenting how to upgrade. Still working on that because I used to have PRO but reverted to CE for simplicity and didn't suffer much loss.

About S3, I like it but I prefer a layered approach, local working, remote #1 in format #1, remote #2 in format #2, even a periodic sync to a 2Tb external drive.

One thing I like about Seafile is that native files are usually synced locally, so if Seafile server or VPS goes down, it's not that difficult (just time) to repopulate a new one. The only issue is Seafile libraries which are "cold", not synced by anyone. But my new SeaClone app gets around that by downloading all libraries as native files.

I am looking into making SeaClone back up Seafile libraries into a S3 destination also, which can then handle the live vs cold vs glacier issue. And maybe different zones. But I want to get the other functions proven out first, including transferring 300Gb of non-cloudron-seafile docs into the cloudron Seafile app.

With SeaClone, a full text search based on Meilisearch and extending SeaClone to handle S3, I am not sure Seafile PRO will be necessary. But will also get around to documenting how a user can upgrade their cloudron Seafile CE to cloudron Seafile PRO.

[girish]

@timconsidine said in Apps for file management/sharing/syncing:

Cubby is also long standing, but doesn't have desktop apps.

I started on an android app - https://git.cloudron.io/cubby/cubby-android/ . It will take some time to get it polished but it will get there.

By desktop app, do you mean a desktop syncer? I connect via webdav on linux and while not a "syncer" , it gives similar functionality. But I am happy to hear opinions on how important a desktop syncer is these days.