LocalAI platform

[eyecreate]

I've got a first version of LocalAI for cloudron. I've only tested some basics and you'll need to allow the app to access any GPUS.

https://git.cloudron.io/eyecreate/localai

https://git.cloudron.io/eyecreate/localai/-/raw/master/CloudronVersions.json?ref_type=heads

[james]

Hello @eyecreate
Apologies, I have simply missed your topic somehow.
I have now added your community app to https://forum.cloudron.io/topic/15172/community-apps

[timconsidine]

@eyecreate well done, thank you, presumably it needs GPU to work efficiently ?

[eyecreate]

I haven't tested it with a gpu(would need to pass device through i think), but yeah, it likely won't work well without one(though ik_llama backend might be good for modern cpus).

[max]

Hi @eyecreate I have installed your localai app, all went well.
I have one issue when I try to connect my LibreChat to it - where do I change the authentication key?

If I test it with I get an expected auth error...
$ curl https://localai.xxxx.net/v1/models
{"error":{"code":401,"message":"An authentication key is required","type":"invalid_request_error"}}

[max]

OK, I think I have worked it out - use the Cloudon CLI...
cloudron env set --app localai.mydomain.com LOCALAI_API_KEY=xxxx
Thanks

[timconsidine]

@eyecreate I tried the community app but everything is just 502.
But I also tried installing their Mac DMG and it could not work with the stated/recommended localhost:8080.

I think it is broken upstream.

Their webpage says run models on your CPU, don't need GPU. Well, it doesn't run on 128Gb RAM modestly used by other apps.

[eyecreate]

Yeah, I've had to report bugs sometimes on hardware compatibility issues.
I've gotten rocm backends working for llama.cpp on the community app, but not others(and something seems off performance wise).

[micmc]

Hi @eyecreate Apparently now the image is not available anymore from the docker repository?
For the moment it refuses to install even after a few retries.

An error occurred during the install operation: Not found: Unable to pull image docker.io/eyecreate/cloudron-localai:v4.10.0. message: (HTTP code 404) unexpected - manifest for eyecreate/cloudron-localai:v4.10.0 not found: manifest unknown: manifest unknown statusCode: 404

Are you still maintaining this package?
Thanks