Feedback Wanted: New Meilisearch File Indexer App – SFTP/S3 & Local Files?

[timconsidine]

I’m currently packaging an application for Cloudron that provides a file content search engine powered by Meilisearch.

N.B. Meilisearch is not an app - it's an API used to create and search indexes. It needs a front end.

My app is a "Search & Indexer UI" that lets you search your documents (PDFs, DOCX, Code, etc.) at Meilisearch speed using indexed data.

While I am thinking about putting parts of this app into others, e.g. Seafile CE with extra search, the vision of the current app is to be a "go to" index of your files, locating files saved in 'wrong' places and answering the question :
"where the hell is that file I used for xxxxx".
So it's a central query point pulling in index info from different sources (index targets) .

[demo data only]

Current Status:
The app is fully functional for indexing files that are:

1. Stored locally within the app’s data directory (not prime focus but possible in some usage scenarios).
2. Mounted via Cloudron Volumes (e.g., external mounts, Hetzner Storage Box etc).
   Multiple index targets can be defined.

A file viewer/downloader (click link to view or download indexed file from where it is actually stored) is in course of being added.

Next :
Before I commence work for the next features, I’d love to get a sense of how you might use this tool to determine where I should focus my development efforts.

I have two specific questions for the community:

1. Direct Remote Indexing (SFTP / S3)
Currently, you can index remote files by mounting them as a Cloudron Volume.

• Question: Would you have a need to index remote locations (via SFTP or S3) without mounting them as a Cloudron Volume?
• Idea: You would simply configure sftp://user:pass@host/path or an S3 bucket directly in the app's settings, and the indexer would crawl it periodically.

2. Local Device Indexing
Many of us have files on our laptops/desktops that aren't synced to the server but that we still want to be searchable alongside our cloud files, especially if you're working on your laptop and can't find or remember where is a file which is actually on your desktop.

• Question: Would you be interested in a small "Local Agent" (a lightweight binary tool) that runs on your Mac/Windows/Linux machine?
• Idea: It would index your specified local folders and securely push the index data to your Cloudron Meilisearch app, making your local files searchable in the same centralized UI, telling you where you put the darned thing.

I’m weighing the complexity of these features against their utility, so any feedback on whether these fit your workflows would be incredibly helpful!

I might well be over-complicating it all, and the current app (focussing on defined Cloudron volumes/mounts) is enough.

Thanks!

[robi]

1. Looks like a more beneficial direction, since not many solutions exist, compared to many for local indexing.

Also, If this will be a App/Service on Cloudron it will need a way to remotely index other local apps installed, if that's the drift I'm seeing.

Otherwise it's an embedded instance per App that has the indexing need.

[timconsidine]

@robi thanks for the feedback

I tend to agree.

The initial vision is a cross-device cross-storage cross-app unified index, and not everyone is comfortable with volume mounts.

Certainly the ability for the unified index to include results from other apps, e.g. a user's Nextcloud deployment, would be good, and it's not really a unified index if certain silos are not included. Need to look into that.

Embedded instance per app is overhead and not a great direction to take, but there might be some cases where a user might want a more targeted and integrated facility (I'm thinking of Seafile CE). Such a user might typically be someone who is much more organised and disciplined than me (who is drowning in files stored in multiple places because that seemed right at the time but I forget within an hour where I put something )

I'm building a road map of other functionality, including the inescapable AI summarisation of docs and indexing.

[necrevistonnezr]

The local agent sounds pretty cool…
And anything that can provide a meaningful search experience for all mails hosted on Cloudron. Search functionality on all self-hosted webmailer‘s is terrible.

[Kubernetes]

I like both options!

[timconsidine]

I decided there was no need of either/or scenario.

So I went ahead and implemented S3/SFTP/WebDav in addition to "local" (meaning actually local to the app and effectively local by means of a mounted volume).

Also I added a desktop tool for user's working devices.

Testing it all out, and will then release it as a custom app.

I decided that a change of app name is appropriate.
It is now "SRTD".
You can see the website for the app at https://srtd.appx.uk

[jdaviescoates]

@timconsidine said in Feedback Wanted: New Meilisearch File Indexer App – SFTP/S3 & Local Files?:

It is now "SRTD".

As in "Sorted"?

[robi]

@jdaviescoates maybe Search Remote Transferred Data?

[timconsidine]

@jdaviescoates yep, that was in my mind
"sorted" as in properly filed and organised
"sorted" as in problem resolved

Best not to put me in charge of any marketing

[timconsidine]

@robi but hey, that might be a lot better

[robi]

@timconsidine oh, ha, I see.. just with the vowels removed.

So britt.

[joseph]

I like it that you went for a pull based approach (i.e the indexer simply pulls the data from various sources). Cubby uses recoll . Do you have any experience with that? Wondering how it compares with meili.

[timconsidine]

@Joseph No, I don't know recoll but will enjoy learning about it.

I just went with Meilisearch because of its general reputation. And it could be a good way to overcome my digital drowning.

I will explore recoll.

[humptydumpty]

Boy, do I need this. Some questions..

• Can we add/lookup other file formats (like 3D files STL, 3DM, OBJ..)?
• I use Cryptomator to encrypt my files locally and on Nextcloud, can it still index/lookup files if I unlock the vault(s)?

[timconsidine]

@humptydumpty thanks for the feedback

• yes, I am planning to add more file formats. The amount of text that can be retrieved/abstracted for indexing might be limited and file-format-specific, but I have several files where I don't need the file content indexed, but I do need to know there is a file of a certain title and location

• the release is still early, so happy to add in suggested features

• Cryptomator : I sed to use something similar, but don't now, so not sure. But my instinctive response is that if the index target (the file storage location) is exposed for trawling and your file system access can display the content, then indexing should be possible. We will only know by trying. I can try to set something up, but if you installed the app, try it out and let me know, and we will see what we can do.

[humptydumpty]

@timconsidine said in Feedback Wanted: New Meilisearch File Indexer App – SFTP/S3 & Local Files?:

your file system access can display the content, then indexing should be possible.

Yes, Windows can search the vault if unlocked.

@timconsidine said in Feedback Wanted: New Meilisearch File Indexer App – SFTP/S3 & Local Files?:

but if you installed the app, try it out and let me know

I don't see the app in CCAI or in the app store. I must have misunderstood you?

[timconsidine]

@humptydumpty

CCAI (open hosted edition) ccai.appx.uk has Zoneminder in the catalogue.
It should be showing penultimate row 2nd element
The catalogue is getting busy, maybe wood for trees scenario. Try a Cmd-F (on Mac - Ctrl-F on Windows ??) and search for 'zone'. - it should show the entry

CCAI-P (closed personal edition hosted on your chosen app domain) needs to import the master catalogue periodically. The default install of CCAI-P probably is using an old or basic catalogue. Press the Import button and accept the suggested source (https://ccai.appx.uk).

There is a glitch currently (on the roadmap to fix) that CCAI-P merges your existing catalogue with the master catalogue but jumbles the order. So the browser Cmd-F / Ctrl-F trick might be useful here.

Good news on Windows being able to read Cryptomator files when unlocked. I guess it would have to do this to be functional. If the file system can find a file and rad its content, then I would expect the Cloudron Meilisearch to work. But let me know your experience !

[humptydumpty]

@timconsidine said in Feedback Wanted: New Meilisearch File Indexer App – SFTP/S3 & Local Files?:

CCAI (open hosted edition) ccai.appx.uk has Zoneminder in the catalogue.
It should be showing penultimate row 2nd element
The catalogue is getting busy, maybe wood for trees scenario. Try a Cmd-F (on Mac - Ctrl-F on Windows ??) and search for 'zone'. - it should show the entry

I can find Zoneminder and ZM alpha. I meant about the meilisearch thing. I thought you had it packaged and wanted to test it with Cryptomator.

[timconsidine]

@humptydumpty Ooops, sorry, brain in dev mode on another project

If you have ZM alpha in your CCAI-P, it means a previous catalogue was imported. You can remove the alpha config by going to File Manager and editing /app/data/apps-config.json.
Remove the "block" for ZM Alpha (it probably won't work now anyway).
To preserve the JSON integrity, ensure you delete the block including the enclosing {} braces, and the closing } brace is followed by a comma , unless it is the last in the list.
This is a bit 'hacky' so will add a UI to CCAI-P so you can do it in browser more safely.

About finding Cloudron Meilisearch in the catalogue :

• ooops, sorry !
• no, it's not shared out yet - my bad
• but also note that during development, I changed the name of the app to "SRTD" (explanation in earlier messages in this topic)
• I will publish the repo tonight and add to CCAI so you can update master your CCAI-P from the master catalogue
• will post here when done