ETA for GPU support? Could we contribute to help it along?
-
@Lanhild said in ETA for GPU support? Could we contribute to help it along?:
I feel like that if GPU support works, having OpenWebUI and Ollama as separate packages would make maintenance easier.
I note this is a bit off topic for the thread... But... It's an interesting idea. Perhaps it depends on your use case.
I can't speak to Cloudron product maintenance of course, only guess. As we're still learning about the relationships between all the components in the stack (Cloudron, Docker, Ubuntu, GPU drivers - just Nvidia data center GPUs in my case at the moment, CUDA, Ollama, OpenWebUI, and throw in server hardware and hypervisors if you're in a virtualized hosting scenario), I have come to understand that complexity of dependencies between components is a real challenge.
Perhaps there are different end-user level configurations that need to be performed between Ollama and OpenWebUI (e.g. stuff like GPU support, single sign-on, access permissions), so it could be a good idea for the product team to separate them from that point of view because one might need more updates and testing than the other, or it might provide more flexibility.
But I wonder, in terms of end (Cloudron) user needs, wouldn't you need to be a pretty advanced user to care? I mean, I can think of several end-user cases where separating Ollama from OpenWebUI gives technical or management or performance benefits, like:
- If you're using your Ollama instance from different endpoints (possibly including outside Cloudron)
- If you want to share Ollama access between apps for performance reasons but separate user and data management on different OpenWebUI instances (e.g. if you have multiple OpenWebUI instances running on one or more Cloudron installations)
- If the cost of hosting resources like storage/compute/RAM is an optimisation concern (e.g. even installing multiple instances of a single app can eat up premium storage space, and sharing compute/RAM of Ollama transactions among multiple apps could have a measurable benefit with more than a few OpenWebUI front ends)
- If you already have a centrally managed non-Cloudron Ollama server but you want Cloudron for OpenWebUI front ends
- If you want to reduce risk of stuff breaking between updates
- And plenty of other stuff along those flexibility lines...
... but otherwise if I'm a regular simple Cloudron user with a GPU installed, I think I just want to one-click-download and have it working without any fuss. I'm guessing (though I don't know) that most Cloudron customers are running at a relatively small scale where simplicity is more important than performance and flexibility. (Please do correct me on that if needed.)
To be clear, it's probably a really good thing to offer in my company's case. But I think we might be in the minority here.
@robw said in ETA for GPU support? Could we contribute to help it along?:
- If you already have a centrally managed non-Cloudron Ollama server but you want Cloudron for OpenWebUI front ends
Hey, that's my case
.Otherwise, I very much think that separating both of the Cloudron packages will be beneficial. Considering all the example use cases you listed, the need is more than justified.
A lot of companies that might deploy Cloudron for its ease of life features don't necessarily have a VPS with a GPU.
Also, (might help you to deepen your Cloudron knowledge) Cloudron packages usually are only one component/application.
Moreover, OpenWebUI is "just" a UI that supports connections to Ollama and isn't affiliated with it. Meaning that Ollama isn't a dependency of it at all.
-
@robw said in ETA for GPU support? Could we contribute to help it along?:
- If you already have a centrally managed non-Cloudron Ollama server but you want Cloudron for OpenWebUI front ends
Hey, that's my case
.Otherwise, I very much think that separating both of the Cloudron packages will be beneficial. Considering all the example use cases you listed, the need is more than justified.
A lot of companies that might deploy Cloudron for its ease of life features don't necessarily have a VPS with a GPU.
Also, (might help you to deepen your Cloudron knowledge) Cloudron packages usually are only one component/application.
Moreover, OpenWebUI is "just" a UI that supports connections to Ollama and isn't affiliated with it. Meaning that Ollama isn't a dependency of it at all.
@Lanhild said in ETA for GPU support? Could we contribute to help it along?:
A lot of companies that might deploy Cloudron for its ease of life features don't necessarily have a VPS with a GPU.
Also, (might help you to deepen your Cloudron knowledge) Cloudron packages usually are only one component/application.
Moreover, OpenWebUI is "just" a UI that supports connections to Ollama and isn't affiliated with it. Meaning that Ollama isn't a dependency of it at all.
Excellent points @Lanhild - you've convinced me.
And there are benefits on the Ollama side too. I would appreciate the benefit in using Cloudron to keep our Ollama installation automatically up to date on its own, for instance.
In fact, given our remaining inability to modify the existing Cloudron OpenWebUI app to run with our GPUs, for our small clients we are now thinking this way - I.e. using Cloudron just for the OpenWebUI component and letting them connect to our separately hosted Ollama. It's a bit less convenient than we were hoping, but at least we'll still have segregated data and user management for each client in OpenWebUI.
So now, I also want a Cloudron OpenWebUI app that does not come with bundled Ollama, so that I can be sure these customers don't hammer our CPUs and get frustrated by a slow user experiences.
-
R robw referenced this topic on
