Feedback Wanted: New Meilisearch File Indexer App – SFTP/S3 & Local Files?

[timconsidine]

@robi thanks for the feedback

I tend to agree.

The initial vision is a cross-device cross-storage cross-app unified index, and not everyone is comfortable with volume mounts.

Certainly the ability for the unified index to include results from other apps, e.g. a user's Nextcloud deployment, would be good, and it's not really a unified index if certain silos are not included. Need to look into that.

Embedded instance per app is overhead and not a great direction to take, but there might be some cases where a user might want a more targeted and integrated facility (I'm thinking of Seafile CE). Such a user might typically be someone who is much more organised and disciplined than me (who is drowning in files stored in multiple places because that seemed right at the time but I forget within an hour where I put something )

I'm building a road map of other functionality, including the inescapable AI summarisation of docs and indexing.

[necrevistonnezr]

The local agent sounds pretty cool…
And anything that can provide a meaningful search experience for all mails hosted on Cloudron. Search functionality on all self-hosted webmailer‘s is terrible.

[Kubernetes]

I like both options!

[timconsidine]

I decided there was no need of either/or scenario.

So I went ahead and implemented S3/SFTP/WebDav in addition to "local" (meaning actually local to the app and effectively local by means of a mounted volume).

Also I added a desktop tool for user's working devices.

Testing it all out, and will then release it as a custom app.

I decided that a change of app name is appropriate.
It is now "SRTD".
You can see the website for the app at https://srtd.appx.uk

[jdaviescoates]

@timconsidine said in Feedback Wanted: New Meilisearch File Indexer App – SFTP/S3 & Local Files?:

It is now "SRTD".

As in "Sorted"?

[robi]

@jdaviescoates maybe Search Remote Transferred Data?

[timconsidine]

@jdaviescoates yep, that was in my mind
"sorted" as in properly filed and organised
"sorted" as in problem resolved

Best not to put me in charge of any marketing

[timconsidine]

@robi but hey, that might be a lot better

[robi]

@timconsidine oh, ha, I see.. just with the vowels removed.

So britt.

[joseph]

I like it that you went for a pull based approach (i.e the indexer simply pulls the data from various sources). Cubby uses recoll . Do you have any experience with that? Wondering how it compares with meili.

[timconsidine]

@Joseph No, I don't know recoll but will enjoy learning about it.

I just went with Meilisearch because of its general reputation. And it could be a good way to overcome my digital drowning.

I will explore recoll.

[humptydumpty]

Boy, do I need this. Some questions..

• Can we add/lookup other file formats (like 3D files STL, 3DM, OBJ..)?
• I use Cryptomator to encrypt my files locally and on Nextcloud, can it still index/lookup files if I unlock the vault(s)?

[timconsidine]

@humptydumpty thanks for the feedback

• yes, I am planning to add more file formats. The amount of text that can be retrieved/abstracted for indexing might be limited and file-format-specific, but I have several files where I don't need the file content indexed, but I do need to know there is a file of a certain title and location

• the release is still early, so happy to add in suggested features

• Cryptomator : I sed to use something similar, but don't now, so not sure. But my instinctive response is that if the index target (the file storage location) is exposed for trawling and your file system access can display the content, then indexing should be possible. We will only know by trying. I can try to set something up, but if you installed the app, try it out and let me know, and we will see what we can do.

[humptydumpty]

@timconsidine said in Feedback Wanted: New Meilisearch File Indexer App – SFTP/S3 & Local Files?:

your file system access can display the content, then indexing should be possible.

Yes, Windows can search the vault if unlocked.

@timconsidine said in Feedback Wanted: New Meilisearch File Indexer App – SFTP/S3 & Local Files?:

but if you installed the app, try it out and let me know

I don't see the app in CCAI or in the app store. I must have misunderstood you?

[timconsidine]

@humptydumpty

CCAI (open hosted edition) ccai.appx.uk has Zoneminder in the catalogue.
It should be showing penultimate row 2nd element
The catalogue is getting busy, maybe wood for trees scenario. Try a Cmd-F (on Mac - Ctrl-F on Windows ??) and search for 'zone'. - it should show the entry

CCAI-P (closed personal edition hosted on your chosen app domain) needs to import the master catalogue periodically. The default install of CCAI-P probably is using an old or basic catalogue. Press the Import button and accept the suggested source (https://ccai.appx.uk).

There is a glitch currently (on the roadmap to fix) that CCAI-P merges your existing catalogue with the master catalogue but jumbles the order. So the browser Cmd-F / Ctrl-F trick might be useful here.

Good news on Windows being able to read Cryptomator files when unlocked. I guess it would have to do this to be functional. If the file system can find a file and rad its content, then I would expect the Cloudron Meilisearch to work. But let me know your experience !

[humptydumpty]

@timconsidine said in Feedback Wanted: New Meilisearch File Indexer App – SFTP/S3 & Local Files?:

CCAI (open hosted edition) ccai.appx.uk has Zoneminder in the catalogue.
It should be showing penultimate row 2nd element
The catalogue is getting busy, maybe wood for trees scenario. Try a Cmd-F (on Mac - Ctrl-F on Windows ??) and search for 'zone'. - it should show the entry

I can find Zoneminder and ZM alpha. I meant about the meilisearch thing. I thought you had it packaged and wanted to test it with Cryptomator.

[timconsidine]

@humptydumpty Ooops, sorry, brain in dev mode on another project

If you have ZM alpha in your CCAI-P, it means a previous catalogue was imported. You can remove the alpha config by going to File Manager and editing /app/data/apps-config.json.
Remove the "block" for ZM Alpha (it probably won't work now anyway).
To preserve the JSON integrity, ensure you delete the block including the enclosing {} braces, and the closing } brace is followed by a comma , unless it is the last in the list.
This is a bit 'hacky' so will add a UI to CCAI-P so you can do it in browser more safely.

About finding Cloudron Meilisearch in the catalogue :

• ooops, sorry !
• no, it's not shared out yet - my bad
• but also note that during development, I changed the name of the app to "SRTD" (explanation in earlier messages in this topic)
• I will publish the repo tonight and add to CCAI so you can update master your CCAI-P from the master catalogue
• will post here when done