Ollama is now available

[nebulon]

Ollama is now packaged as a standalone app.

The app package is as usual still published as unstable until we get some feedback how it works and iron out any initial issues.

Forum category is at https://forum.cloudron.io/category/212/ollama so please report any issues there
The docs will be at https://docs.cloudron.io/packages/ollama/
Package repo is https://git.cloudron.io/packages/ollama-app

[firmansi]

Finally...hopefully it can be detached with openwebui in next update

[nebulon]

Yes that was the plan, we will remove it from OpenWebUI once the standalone package gets stable.

[abuyuy]

Is it able to use graphics card?

[james]

Hello @abuyuy
Currently, the Ollama app does not include the capability to access vaapi devices (gpus).
We can add this capability to the Ollama app but how to configure it needs to be researched then.

[abuyuy]

Thanks James for the quick answer.
I think having GPU support for Ollama would be tremendously helpful for the performance and usability of the app.

[nebulon]

@abuyuy yes fully agree, we will first focus on getting the app to a stable package state and then investigate what all is required for hardware support.

[ekevu123]

How do you intend to get the GPU to the server? With most public hosting plans, this is quite expensive? I guess that is only relevant for people using their own hardware?

[firmansi]

Eventhough it's out of context of this thread but i see that the Cloudron team already start the detachment process of Ollama and OpenWebui

[robw]

Firstly, this is wonderful news. I see that your Ollama package appears to include an authentication capability, which is something that vanilla Ollama is missing. When coupled with Cloudron's app management features, that makes it extremely helpful.

However... Without GPU support or crazy fast CPUs and a big load of fast RAM, isn't Ollama self hosting almost useless?

Note - Here's an old thread where we discussed some related topics.

[robw]

@james In our case (we're have some old enterprise Nvidia GPUs), we found that installing the Nvidia drivers and CUDA toolkit is ultimately not difficult once you know what to do. Our problem with making that useful on Cloudron (per this old thread) was that we couldn't start Cloudron's OpenWebUI package with GPU support. So even though we had our GPU hardware working on a Cloudron server, we could never use it with a Cloudron instance of OpenWebUI/Ollama.

I suspect the process of installing the Nvidia drivers and CUDA toolkit is generic enough that it could be supported by Cloudron somehow. However we don't have any GPUs from other enterprise brands (e.g. AMD, Intel) so we never tried that, and I imagine supporting drivers for consumer GPUs is an impossible maze of complexity.

We would LOVE it if you could support the Nvidia GPU use case as a basic option.

[joseph]

@robw this package is simply the first step. second step is to enable GPU/VAAPI support in docker (this requires some complex automation). @james is looking into this.

[nebulon]

The package is now marked as stable after some internal reorganization, will lock this thread in favor of the dedicated forum section.