Feedback Wanted: New Meilisearch File Indexer App – SFTP/S3 & Local Files?

[timconsidine]

I’m currently packaging an application for Cloudron that provides a file content search engine powered by Meilisearch.

N.B. Meilisearch is not an app - it's an API used to create and search indexes. It needs a front end.

My app is a "Search & Indexer UI" that lets you search your documents (PDFs, DOCX, Code, etc.) at Meilisearch speed using indexed data.

While I am thinking about putting parts of this app into others, e.g. Seafile CE with extra search, the vision of the current app is to be a "go to" index of your files, locating files saved in 'wrong' places and answering the question :
"where the hell is that file I used for xxxxx".
So it's a central query point pulling in index info from different sources (index targets) .

[demo data only]

Current Status:
The app is fully functional for indexing files that are:

1. Stored locally within the app’s data directory (not prime focus but possible in some usage scenarios).
2. Mounted via Cloudron Volumes (e.g., external mounts, Hetzner Storage Box etc).
   Multiple index targets can be defined.

A file viewer/downloader (click link to view or download indexed file from where it is actually stored) is in course of being added.

Next :
Before I commence work for the next features, I’d love to get a sense of how you might use this tool to determine where I should focus my development efforts.

I have two specific questions for the community:

1. Direct Remote Indexing (SFTP / S3)
Currently, you can index remote files by mounting them as a Cloudron Volume.

• Question: Would you have a need to index remote locations (via SFTP or S3) without mounting them as a Cloudron Volume?
• Idea: You would simply configure sftp://user:pass@host/path or an S3 bucket directly in the app's settings, and the indexer would crawl it periodically.

2. Local Device Indexing
Many of us have files on our laptops/desktops that aren't synced to the server but that we still want to be searchable alongside our cloud files, especially if you're working on your laptop and can't find or remember where is a file which is actually on your desktop.

• Question: Would you be interested in a small "Local Agent" (a lightweight binary tool) that runs on your Mac/Windows/Linux machine?
• Idea: It would index your specified local folders and securely push the index data to your Cloudron Meilisearch app, making your local files searchable in the same centralized UI, telling you where you put the darned thing.

I’m weighing the complexity of these features against their utility, so any feedback on whether these fit your workflows would be incredibly helpful!

I might well be over-complicating it all, and the current app (focussing on defined Cloudron volumes/mounts) is enough.

Thanks!

[robi]

1. Looks like a more beneficial direction, since not many solutions exist, compared to many for local indexing.

Also, If this will be a App/Service on Cloudron it will need a way to remotely index other local apps installed, if that's the drift I'm seeing.

Otherwise it's an embedded instance per App that has the indexing need.

[timconsidine]

@robi thanks for the feedback

I tend to agree.

The initial vision is a cross-device cross-storage cross-app unified index, and not everyone is comfortable with volume mounts.

Certainly the ability for the unified index to include results from other apps, e.g. a user's Nextcloud deployment, would be good, and it's not really a unified index if certain silos are not included. Need to look into that.

Embedded instance per app is overhead and not a great direction to take, but there might be some cases where a user might want a more targeted and integrated facility (I'm thinking of Seafile CE). Such a user might typically be someone who is much more organised and disciplined than me (who is drowning in files stored in multiple places because that seemed right at the time but I forget within an hour where I put something )

I'm building a road map of other functionality, including the inescapable AI summarisation of docs and indexing.

[necrevistonnezr]

The local agent sounds pretty cool…
And anything that can provide a meaningful search experience for all mails hosted on Cloudron. Search functionality on all self-hosted webmailer‘s is terrible.

[Kubernetes]

I like both options!