AnythingLLM - AI business intelligence tool

[humptydumpty]

@timconsidine said in AnythingLLM - AI business intelligence tool:

Does anyone want it ?

I was interested until chatgpt told me it requires a gpu to work reasonably fast or else it falls back to the cpu. Unless it's mistaken and it would work okay for text prompt on older servers/cpus (6th gen intel)?

[timconsidine]

@Kubernetes ha ! I wish. My good friend TRAE helps me a lot, now with help of context7 and Sequential Thinking MCPs.

Amazing how stupid they can be at times, though !

[timconsidine]

@humptydumpty could be a good point

But I am NOT running on a GPU.
I just have a good Hetzner server (128Gb, good CPU).

I guess it's a limbo dance trial and error - how low can you go ?

I have a 'spare' testing Cloudron instance - very modest specs - just a mini PC - will give it a go over the weekend if I have a chance.

[humptydumpty]

@timconsidine My home-servers are 6th or 7th gen Intel (i5-6500t or i5-7500t) both with 32GB RAM. I'll give it a try if you make it public on CCAI. What's the worst that could happen ?

[timconsidine]

@humptydumpty
I will tidy it up (remove some debug code) and let you know when on CCAI.

[robi]

The performance depends on the model you load and run.

One guy found out he could run a small model (gemma 3) on his CPUs as multiple agents which fronts other API based models and agents who do heavier lifting.

[joseph]

@timconsidine you are on a roll!

[timconsidine]

@joseph trying to get stuff done which I need to be productive on a big (non-Cloudron) project (RAG in this case)

and that AppWishList list frustrates me

[micmc]

@timconsidine said in AnythingLLM - AI business intelligence tool:

@ joseph trying to get stuff done which I need to be productive on a big (non-Cloudron) project (RAG in this case)

and that AppWishList list frustrates me

Amazing work mate, all the way. Frustrated by the list, I get yah

I know the app well. As it's been said the native app is absolutely amazing so I run it desktop for some purpose, but I also run in testing an instance on docker, online.

I'd be glad to test your cloudron version on a i7 dedicated server with 128GB of RAM and no GPU at all, from Hetzner so we can confirm it runs well. I guess it would run well just as Open WebUI it all depends on the models you use.
Cloudron CLI info would suffice, if you want to shoot'em to me when you've a minute my 'speedy' on the run
Cheers!

[timconsidine]

@micmc someone with knowledge of the app would be very helpful to test it out - I'm not so familiar so half the battle is finding out what to test

I think I held off pushing out the repo and docker image because of a couple of queries, and I find myself pulled (well, my fault entirely) between completion of 5-6 apps.

Will try to get this out the door shortly.
Any feedback you then have will be gratefully received.