Cloudron Forum

@nebulon Let me use "vmstat 1" as you suggested and see what is really going on.

@nebulon Great point about the VPS. I am going to try "monitoring" two different Cloudrons at different VPS providers. Let's see if there is a similar pattern (perhaps something related to Clourdon) or if one Cloudron is having API issues, then that would suggest a VPS provider issue. In any case and on all Cloudrons, I will check the box.log to see if there is something happening on the Cloudron when API issues appear.

Independent of the API/heartbeat issues, what is your recommendation on managing the swap file and it's size?

@nebulon At times I see that the Cloudron is not responsive (browser timeouts, which could be something other than a busy Cloudron server). I am also having an issue when using external monitors to check the health of my Cloudron (Uptime Robot, BetterStack). Attempts to use the API health check to return Cloudron version sometimes fails (causing Uptime or BetterStack to issue an alert). The same problem happens using the appid API call to check my Kuma instance. Again, I am not sure why this is happening or if it relates to the swapfile and performance. The GET failures are intermittent, but frequent enough to cause me to disable these alerts.

Given the CPU/RAM/Disk of this server, I would not expect any of this to be an issue. I am not seeing anything in the box log that might suggest a problem. An article I was reading prompted me to look at the swapfile and utilization. Inside Cloudron, the monitoring chart seems to show a 50/50 split between RAM and swapfile. Ideally, Cloudron would use close to 100% RAM, but if subsystems dictate a 50/50 split, then perhaps I should increase from an 8GB swapfile to something much larger (I have enough disk to accommodate this easily).

Theoretically, suppose I allocate 1GB to /apps.swap? How would that impact Cloudron performance? Would applications crash? Wait for available swapfile RAM? Just trying to figure out how to best allocate extra resources to get better performance.

My server has a lot of RAM and I would prefer to leverage that rather than disk (swap). So I reduced the "swappiness" factor from 60 (Ubuntu default) to 10 expecting to reduceswapping. I can see that the server is using that value. That being said, it doesn't look like Cloudron honors swappiness. How can I fix this?

It also looks like the standard /swapfile was not being used (so I removed it), but Cloudron uses /apps.swap.

If it can't be "fixed", do I need to increase the size of my swapfile? (When I increased /apps.swap from 4GB swap to 8GB, it looks like Cloudron sucked up the extra swap.) What is the right value for max performance (assuming I have extra disk space to spare) as I add more apps?

@joseph Thanks for this update. I see no point (for me) in going further with Postiz. I would support looking for a better alternative.

Decided to implement Postiz and after working through the config and options, I realized that the app is of marginal use on Cloudron. Here are the details:

(1) You can enable OIDC which is great. But if you disable registration, you are unable to invite any other users to your "workspace".

(2) If using OIDC, potentially each Cloudron user could have their own workspace, but you would have to enable registration, add the user, and then disable the user.

(3) Assuming you follow the path in #2, there is no admin interface to see the users in your instance, leaving you with the potential that someone unknown has signed up and is using your Postiz instance.

(3) If you use the app's own authentication (not OIDC), you can invite other users, but can only have a single workspace per domain. (see #4)

(4) #3 would not be a problem as you can define a workspace for each Cloudron domain you wish, except that the Docker container is massive (4GB) due to an unoptimized node_modules. So the cost of an instance per domain is 4GB of disk and 1GB of RAM. A lot of resources wasted.

If you are social media team of 1 person and need only one workspace on your server, I can see where Postiz would work. But with Cloudron's strength being a multi-domain architecture, Postiz seems to be lacking in many areas.

Any other opinions?

@timconsidine AI "guessed" 41-57 hours. So reality was just above the upper estimate. Also, not sure if AI "guessed" a PRO version instead of Community versions (which is what was possible).

On the topic of automated backup, if you store your SeaFile objects on Amazon S3, it is possible with bucket policies to replicate objects to a cheaper storage tier. No need to extend the application. And then you can have "deep storage" files delete automatically after a certain period of time. I don't know if some of the S3-compatible services like iDrive E2 and BackBlaze offer this capability but there are some interesting possibilities.

Finally, read an article that gave me pause about backups. A small company lost millions in revenue because their cloud provider (who shall be anonymous) went down and their backups were in the same availability region (which was part of the outage). This is making me think that perhaps duplicate backups (whether S3 objects or VPS snapshots might be worth replicating in another availability zone.

@timconsidine Gut feel, how much more (as a percent)? Just trying to calibrate AI as an estimating tool.

@james I like your thinking. This might allow for better hardware utilization on smaller servers. @svtx I get that some TURN queries are legitimate. But I would think that a service utilizing TURN would broadcast this to others so that they can "connect". Since I am not broadcasting, I am not sure why people are probing for TURN connections. I guess I am jaded by the thousands of unauthorized SSH attempts I see on all my servers each day. So my bias has shifted from "trust, but verify" to "distrust and verify when someone complains".

@timconsidine Understood. But as I recall, AI was projecting a 40+ hour effort. Perhaps the last few tasks will take a lot of time. Been there!

@timconsidine You are impressive my friend. Can't believe you were able to package this so quickly. Just curious: where do you think the AI estimates are inflated? Perhaps Tim is 4x faster than others:)

And you are 100% correct about the <= 3 scenario. I missed that.

@timconsidine I remember reading about a PRO license for small business (up to 9 seats) for $100 per year (self-hosted). So not free technically, but essentially free given the value provided.

@tim What I like about S3 (and compatibles) is the "five 9s" reliability of object storage and unlimited expansion capability. I guess I grew tired of managing local storage...

Seafile ongoing maintenance effort seemed concerning (assuming it's realistic). Curious about what Cloudron team thinks about maintaining either.

Also, Pydio seemed to earn some kudos for a lightweight, high-performing tech stack. The analysis seemed to suggest it really leverages the S3 API - which is solid at this point.

@timconsidine So I had a few minutes and "tasked" my right-hand assistant with the job of evaluating options. Here was her assignment:

Research all open source apps that can be used for file storage similar to Box.com and Dropbox. App must:

• Use S3-compatible storage as backend for files (object) storage.
• Support infinite versions.
• Have good iOS and Android mobile apps for sharing, moving and copying files.
• Docker installation option.

In addition, for those meeting all the above criteria, analyze the tech stack and determine whether packaging the app for Cloudron would require small, medium or large amount of work.

Also assess the frequency of updates and whether that would become a large burden for Cloudron maintainers to keep up.

Here were the results:

Executive Summary

S3 Storage Analysis

Cloudron Packaging

She missed that NextCloud was already packaged, but no one is perfect:)

Bottom line: Pydio Cells looks very viable:)

@joseph What are your thoughts from Team Cloudron's perspective?

@timconsidine I was thinking much the same. Currently using Box.com (good, expensive, excellent mobile app, NO LINUX CLIENT) and was wondering about whether Seafile could be packaged for Cloudron. I think you answered that neatly already.

Because of Cloudron's architecture, I think the ability to back up metadata about files from a database (built-in) and using an S3-compatible back-end store for files would be an ideal solution.

For me, a strong mobile app is a necessity. There have been countless times when someone asked for a file, and with the Box iOS app, I was able to get it to the recipient instantly.

I will keep looking. Was hoping that Pydio could be the one, but sounds like it's not quite ready.

Since Cloudron already manages "allowed ports" internally, I think that adding TURN server ports to this list is a necessary security feature. Here are the details:

Background:

Several Cloudron users have reported that unwanted (hacking?) attempts are being made to connect to their Cloudron's TURN server despite the fact that no installed apps utilize TURN.

Server resources (256MB RAM + application logs) are being wasted when no app needs an operational TURN server.

Managing this external to the Cloudron server via firewall or proxy leaves the potential for a support issue when a user adds an app that needs TURN, but forgets to update their firewall and enable the ports. (Note: Also, this solution just blocks the connection, but still wastes resources).

Proposal:

Have Cloudron handle TURN server management (resources, ports) internally with the following logic:

(1) If an app requires TURN server access (it should be declared in the app manifest). If that occurs, then the TURN server container should be "brought up" if not already enabled, resources (memory) deployed according to the configuration, and TURN ports permitted in the firewall.

(2) If no apps use TURN, then the server should be disabled, ideally, the container disabled, and TURN ports blocked by the internal Cloudron firewall automatically.

Perhaps a "first step" would be during Cloudron boot, to disable TURN ports (firewall) if no app needed TURN, leaving the container operational as is. This would accomplish the needed security and with no connections being possible, the actual utilization of the RAM should be almost 0.

Everyone, please feel free to add/delete/modify as you see fit!

@girish @dbruehlmeier Will make a Feature Request thread.

@joseph is this a security enhancement that the Cloudron team can add to a future maintenance release? I would prefer to have Cloudron maintain control of the TURN service so it could enable it based on installed dependencies or leave it off if there are none. Much safer and won't create support issues if someone has disabled TURN ports themselves and then forgets to manage this properly externally.

@joseph What I see appears to be "normal" attempts to connect to a TURN server. The problem is that there is no application "publishing" my TURN server to make it usable by potential WebRTC connections. So these are people trying to leverage a misconfigured TURN server or a software vulnerability. In my mind, it's no different than people probing to SSH to a server - hence the reason to use Fail2Ban and other tools to restrict this.

Since Cloudron manages "allowed ports" internally, I think that TURN server ports should be added to that list as follows:

(1) If an app requires TURN server access (it should be declared in the app manifest), then TURN ports should be opened)
(2) If no apps use TURN, then those ports should be closed by CLoudron automatically.

@girish @nebulon @james I am seeing a flood of unauthorized attempts to connect to the TURN server on each Cloudron. I have no apps installed that are using TURN. It seems like a potential security vulnerability and certainly a waste of CPU/RAM (256MB Cloudron min)/Disk resources. One TURN log was already up to 50MB! In the absence of a "switch" in Cloudron to disable this, I wanted some advice on two temporary solutions:

(1) docker stop turn coupled with docker update --restart=no turn

(2) Adding a crontab entry: @reboot /usr/bin/docker stop turn

Option 1 prevents the container from starting. Option 2 allows it to start, but stops it when rebooting. @d19dotca Thanks for starting my thinking on this!

Option 1 seems better, but I wanted an expert opinion on the consequences of using either.

Lastly, I guess I will need to be aware that installing any apps that require the TURN service could fail (unless I enable the TURN container once again). If I forget and install an app like Jitsi, will Cloudron restart the TURN container and revert the --restart=no option?