@girish I tried with this in Cloudron Crontab:
0 * * * * mongosh -u "${CLOUDRON_MONGODB_USERNAME}" -p "${CLOUDRON_MONGODB_PASSWORD}" ${CLOUDRON_MONGODB_HOST}:${CLOUDRON_MONGODB_PORT}/${CLOUDRON_MONGODB_DATABASE} --eval "const collection = db.getCollection('rocketchat_apps_logs'); const toDelete = collection.find().sort({ _createdAt: -1 }).skip(1000); toDelete.forEach(doc => collection.deleteOne({ _id: doc._id }));"
It seems to work.
Update: Fixed the schedule to happen once per hour.
Hello @girish
I did use similar backups and then the rocketchat_apps_logs.bson file got much smaller.
However it only took 3 days and it was too big once again.
So I was wondering if that file has anything to do with Cloudron backups rocketchat_apps_logs.bson as it seems to be created then. But as I said I disabled Cloudron backups for Rocket.chat and these files in /var/lib/docker/volumes/569e4faada8ec760513bad87aa4c494f4792ebf6e5ef0dbd3f9f539d8a0b0ab6/_data/e95dbcf8-8480-47a3-834f-ea3dfbede698 still got created tonight.
Thanks,
Martin
Hello @girish
We run into the same issue with rocketchat_apps_logs.bson being too large.
It's stored in the /var/lib/docker/volumes/569e4faada8ec760513bad87aa4c494f4792ebf6e5ef0dbd3f9f539d8a0b0ab6/_data/e95dbcf8-8480-47a3-834f-ea3dfbede698 folder.
I though that if we disable Cloudron backups for Rocket.chat these files will stop being created but it seems it did not help.
The problem is that the rocketchat_apps_logs table suddenly gets thousands of rows every day and the rocketchat_apps_logs.bson backup file that is created for it is much bigger than the database size of the table. I got about 10 GB file for 2 GB of Mongodb table size
I tried to change the "Apps' Source Package Storage type" in Rochat.chat as you suggested.
Thanks,
Martin
Hello Cloudron support,
I found that our Grav app is 5 GB. I had a look with the console and it's all in /app/data/backup - it contains 167 backup files, each one is 32 MB.
I'm not sure what's creating these backups, we do not use any cronjob for that app.
Could you please make sure that folder is cleaned up properly?
Thanks,
Martin
Hello Cloudron support,
Our Uptime Kuma always fails to backup almost every day.
It has 4.7 GB of data showing in "Storage" tab of the app settings. I checked the app files and I see there is a single kuma.db file which is 4.7 GB. It has data since May 2022, which does not seem like too much.
Uptime Kuma also has a "Shrink Database" button which seems to deal with the kuma.db-wal file which is just 5.48 MB in our case.
When we try to backup the app, it always stops progressing at about 1138MB:
2024-08-08T23:00:02.161Z box:taskworker Starting task 12473. Logs are at /home/yellowtent/platformdata/logs/tasks/12473.log
2024-08-08T23:00:02.266Z box:tasks update 12473: {"percent":1,"message":"Backing up uptime.foliovision.net (1/13)"}
2024-08-08T23:00:02.281Z box:tasks update 12473: {"percent":7.25,"message":"Snapshotting app uptime.foliovision.net"}
2024-08-08T23:00:02.302Z box:services backupAddons
2024-08-08T23:00:02.302Z box:services backupAddons: backing up ["localstorage"]
2024-08-08T23:00:02.303Z box:backuptask snapshotApp: uptime.foliovision.net took 0.022 seconds
2024-08-08T23:00:02.304Z box:tasks update 12473: {"percent":7.25,"message":"Uploading app snapshot uptime.foliovision.net"}
2024-08-08T23:00:02.306Z box:shell backup-snapshot/app_096b330c-bd95-44e1-af1b-cafd8e1cbdc2 /usr/bin/sudo -S -E --close-from=4 /home/yellowtent/box/src/scripts/backupupload.js snapshot/app_096b330c-bd95-44e1-af1b-cafd8e1cbdc2 tgz {"localRoot":"/home/yellowtent/appsdata/096b330c-bd95-44e1-af1b-cafd8e1cbdc2","layout":[]}
2024-08-08T23:00:18.007Z box:tasks update 12473: {"percent":7.25,"message":"Uploading backup 28M@3MBps (uptime.foliovision.net)"}
2024-08-08T23:00:28.004Z box:tasks update 12473: {"percent":7.25,"message":"Uploading backup 59M@3MBps (uptime.foliovision.net)"}
2024-08-08T23:00:38.002Z box:tasks update 12473: {"percent":7.25,"message":"Uploading backup 92M@3MBps (uptime.foliovision.net)"}
[...]
Aug 06 00:43:25 box:tasks update 12446: {"percent":7.25,"message":"Uploading backup 1138M@0MBps (uptime.foliovision.net)"}
[...]
Aug 06 01:00:00 box:tasks update 12446: {"percent":7.25,"message":"Uploading backup 1138M@0MBps (uptime.foliovision.net)"}
[...]
2024-08-09T12:24:20.156Z box:tasks update 12473: {"percent":7.25,"message":"Uploading backup 1146M@0MBps (uptime.foliovision.net)"}
I tried to use the "Resources" tab and increase the memory from 768 MB to 1.5 GB. Then I triggered a manual backup in it's "Backups" tab. Even with that it stopped backing up at 1147M.
It seems it actually did not use more than 245 MB of Memory, even if It look at last 30 days in "Graphs" tab. So it's actually not using all that much memory.
Please let us know how to troubleshoot Uptime Kuma backup issues. 4.7 GB might seem like a lot of data, but it's just 2 years and Uptime Kuma does not seem to run out of memory for backups.
Thanks,
Martin
Hello girish,
I tried that command and got:
May 13 12:52:45 FV-Cloudron systemd[1]: Started /tmp/box-1040295979/scripts/installer.sh.
May 13 12:52:45 FV-Cloudron systemd[1]: cloudron-updater.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
May 13 12:52:45 FV-Cloudron systemd[1]: cloudron-updater.service: Failed with result 'exit-code'.
I run it manually and I see it needs Ubuntu 20.04 or above. Too bad it did not show that error on Cloudron web interface, it would save us a lot of trouble.
Thanks,
Martin
Our Cloudron has a huge 16 GB /tmp folder.
I see there are about 76 folders like this:
drwxr-xr-x 9 yellowtent yellowtent 4096 Aug 29 2023 box-929500886/
drwxr-xr-x 9 yellowtent yellowtent 4096 Aug 29 2023 box-940153435/
drwxr-xr-x 9 yellowtent yellowtent 4096 Aug 29 2023 box-954421980/
And 73 more. All of them have that "Aug 29 2023" date.
When I run service cloudron-updater status it seems one of them is still used:
● cloudron-updater.service - /tmp/box-243670221/scripts/installer.sh
Loaded: loaded (/run/systemd/transient/cloudron-updater.service; transient)
Transient: yes
Active: failed (Result: exit-code) since Mon 2024-05-13 03:34:36 UTC; 7h ago
Process: 11966 ExecStart=/tmp/box-243670221/scripts/installer.sh (code=exited, status=2)
Main PID: 11966 (code=exited, status=2)May 13 03:34:36 FV-Cloudron systemd[1]: Started /tmp/box-243670221/scripts/installer.sh.
May 13 03:34:36 FV-Cloudron systemd[1]: cloudron-updater.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
May 13 03:34:36 FV-Cloudron systemd[1]: cloudron-updater.service: Failed with result 'exit-code'.
Can these /tmp/box-* folders be removed?
Thanks,
Martin
Thank you for the suggestion. We are not noticing issues wit Rocket.chat, so the issue might be with Uptime Kuma or its database.
Unfortunately the issue is still present. It was only fast for a while.
I tried to restart the whole server, but it did not fix the issue. The server is not overloaded and there is plenty of free memory.
The app logs show no errors or warnings. It reports memory usage of 110 MB.
I restarted it again and now it works and it's fast again.
It's weird, did anyone experience anything similar with Cloudron and/or Uptime Kuma?
I tried to modify some of the core Uptime Kuma files in its /app/code/ folder. It was in the Rocket.chat notifications file.
The change did not make any effect, so I tried restarting it.
I was shocked to see the empty list of monitors. It's as if the whole database has disappeared.
However then, couple of minutes later, the list got populated.
I thought it could be a damage done by some of my modifications. So I just restored the backup from yesterday.
But I'm still running into the issue and currently the monitors are not loading at all.
Did anybody experience issues with Uptime Kuma after restart?
Hello,
we are running in an issue where our Rocket.Chat instance is unable to properly send messages after about 20 hours of uptime. Whenever I try to send a message to somebody the message appears gray, indicating that it was not yet sent. The user does actually get the message though, but it impossible to chat like that.
I wanted to check some log files, but I only found that Rocket.Chat logs errors to stdout here: https://github.com/RocketChat/Docker.Official.Image/issues/16 So I tried to run the command, but it have me the error:
# docker logs e5648bb067fe
Error response from daemon: configured logging driver does not support reading
I tried to attach to the docket stdout, but nothing would ever appear:
docker attach e5648bb067fe
I tried to inspect the files on docker too by logging in:
docker exec -it -u root e5648bb067fe /bin/bash
I tried to check the Mongo containers as well:
# docker exec -it -u root 48f5e963462c /bin/bash
Once there I was able to connect to the database:
root@mongodb:/# mongo
But then even basic command like list of tables failed:
rs0:PRIMARY> use rocketchat
switched to db rocketchat
rs0:PRIMARY> show tables;
Warning: unable to run listCollections, attempting to approximate collection names by parsing connectionStatus
Could you please provide any insight?
Thanks,
Martin