Hello everyone,

Thank you all for your support and suggestions throughout this process. I’m happy to report that the issue has been resolved.

After analyzing the resources and following the recommendations provided, we decided to upgrade the AWS instance from t3.large to t3.xlarge. This upgrade provided additional CPU and RAM, which addressed the resource limitations that were causing the server to become unresponsive.

Since the upgrade, we have not experienced any further downtime or performance issues. We can confirm that the problem was related to resource constraints, specifically the depletion of CPU credits and the high reliance on swap memory.

Thank you again for your help and guidance.

Best regards,
Felipe Rubilar

Hi @james and @joseph,

Cloudron Performance Issues Analysis

Thank you for your responses and guidance. I've gathered additional information and performed a detailed analysis of the system resources, as you suggested. Here's what I've found:

1. Logs and Observations

• When the apps become unresponsive, the server itself is also unreachable (cannot ping or SSH into it)
• This confirms that the issue is likely on the server side
• I've attached screenshots of the CPU and memory usage graphs for the last 6 hours, which include the period leading up to the most recent downtime

2. Resource Analysis

CPU Usage:

• The CPU usage shows frequent spikes, with a peak of 169.2% (see attached graph)
• Most of the CPU usage does not seem to come from the applications themselves (e.g., GitLab and Mesa de Ayuda have low CPU usage)
• Since the instance is a t3.large, which relies on CPU credits, I suspect that these spikes might be depleting the credits, causing the instance to slow down or become unresponsive

Memory Usage:

• The memory usage is consistently high, with 7.18 GiB of RAM used out of 8 GiB
• GitLab alone consumes 4.63 GiB, and the system is actively using swap (4.12 GiB)
• The reliance on swap could be degrading performance, especially during high memory demand

Nginx Warnings:

• I noticed several ssl_stapling warnings in the logs for the SSL certificates of my apps
• While these are just warnings, I'm unsure if they could be contributing to the issue

3. Possible Root Cause

Based on the resource analysis, I believe the main issue might be resource limitations:

CPU Credits:

• The high CPU usage might be depleting the CPU credits of the t3.large instance, leading to performance degradation

Memory Constraints:

• The high memory usage and reliance on swap suggest that the instance might not have enough RAM for the current workload

4. Next Steps and Questions

I'm considering upgrading the instance to address these limitations, but I'd like to confirm if this is the root cause of the downtime. Specifically:

• Does this analysis align with the symptoms of the instance becoming unresponsive?
• Would upgrading to a more powerful instance (e.g., t3.xlarge, m5.large, or m5.xlarge) resolve the issue?
• Are there additional steps I should take to optimize the current setup or gather more diagnostic information?

Attachments

• CPU and Memory Usage Graphs
• System Info Screenshot

---

Thank you again for your help! I look forward to your insights.

Best regards,
Felipe Rubilar

Dear Support Team,

For the past three weeks, we have been experiencing recurrent issues with our Cloudron instance. Specifically, the system goes down and all the applications hosted on it become unavailable. This has been happening 1-2 times per week during this period.

Each time this issue occurs, we are forced to restart the server on AWS to restore the services and make the applications accessible again. This situation is significantly impacting our organization, as our team members are unable to work with the tools hosted within the Cloudron container during these downtimes.

We have monitored the server resources (CPU, RAM, storage, etc.), and everything appears to be within normal ranges. Unfortunately, we do not have any logs or additional details to identify the root cause of these failures.

We kindly request your assistance in investigating and resolving this issue.

We appreciate your prompt support and look forward to your guidance to prevent further disruptions.

Best regards,
Felipe Rubilar
Cloud Engineer

Hi Nebulon,

I wanted to let you know that I have installed Thunderbird and am currently synchronizing my mailbox. So far, I haven't encountered any issues, and I will inform you if anything arises.

Additionally, I would like to inquire if there is a storage limit for mailboxes on Cloudron, as I currently have around 500,000 emails in my mailbox, which serve as a backup for client communications.

Thank you for your assistance.

Best regards,

Hello Cloudron Support Team,

I am experiencing an issue with accessing my mailbox configured on Cloudron through the Webmail application. When attempting to log in, I encounter an error message stating "Folders error: Error."

Could you please assist me in resolving this issue? Any guidance or support would be greatly appreciated.

Thank you for your help.

Best regards,

Hi Nebulon,
Thank you for your assistance regarding the backup issue for the mail application. Based on your suggestion and the error logs, I have decided to increase the memory limit for the backup process to 4.5 GiB.

Here is the updated configuration:

Memory Limit: 4.5 GiB
Upload Part Size: 128 MiB

I will monitor the backup process after this change to see if it resolves the issue. I have attached a screenshot of the updated settings for your reference.

If the problem persists, I will provide additional logs and system metrics for further analysis.

Thank you for your support.

Best regards,
Felipe Rubilar

Dear Cloudron Support Team,

I am experiencing a recurring issue with the backup tasks on my Cloudron instance. The backup task for the mail application is consistently crashing, and I am unable to determine the cause or find a solution.

Details:

• Error Log Excerpt:

  2024-07-23T09:11:39.794Z box:shell backup-snapshot/mail: /usr/bin/sudo -S -E --close-from=4 /home/yellowtent/box/src/scripts/backupupload.js snapshot/mail tgz {"localRoot":"/home/yellowtent/boxdata/mail","layout":[]} errored BoxError: backup-snapshot/mail exited with code null signal SIGABRT
  2024-07-23T09:11:40.020Z box:taskworker Task took 11497.389 seconds
  2024-07-23T09:11:40.032Z box:tasks setCompleted - 11410: {"result":null,"error":{"stack":"BoxError: Backuptask crashed\n    at runBackupUpload (/home/yellowtent/box/src/backuptask.js:163:15)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async uploadMailSnapshot (/home/yellowtent/box/src/backuptask.js:404:5)\n    at async backupMailWithTag (/home/yellowtent/box/src/backuptask.js:452:5)\n    at async fullBackup (/home/yellowtent/box/src/backuptask.js:510:26)","name":"BoxError","reason":"Internal Error","details":{},"message":"Backuptask crashed"}}
  

Current Configuration:

• Memory Limit: 2 GiB
• Upload Part Size: 128 MiB

Issue:

I have not made any changes to the default configuration settings. The backup process crashes with a SIGABRT signal, indicating a critical error that the process could not handle.

Request:

Could you please assist me in diagnosing and resolving this issue? I would like to know the specific changes or configurations I should apply to prevent this problem from recurring and to ensure the stability of my Cloudron instance.

Thank you for your support.

Best regards

Thank you, Girish. I have already opened the issue at the link you sent me (https://github.com/Leantime/leantime/issues). I will be attentive to your instructions to resolve this issue.

Regards,

Hi Girish,

I tried again from another browser and the issue persists.

I will send you the step-by-step process I follow to download the documents and the error message I receive.

I log into Leantime and navigate to the projects:

Then I select an activity:

I try to download any of these files associated with the activity:

But when I attempt to download:

I encounter the following error:

I have already tried with another browser and the problem persists.

Hi @nebulon,

In the document download panel, I only see the option "download." I am attaching evidence:

And the issue persists...

Thank you, I am attentive to your instructions.

Hello,

I hope you're having a great day. I'm reaching out to you regarding the following matter.

On Thursday, 14/03, users of our Leantime tool, which is hosted on our Cloudron platform, reported encountering issues with opening/downloading files from the application. Please find attached a screenshot illustrating the problem.

We kindly request your assistance in resolving this matter promptly to ensure uninterrupted user productivity.

We look forward to hearing from you.

Best regards,