mysterious tootctl media remove task

[nichu42]

A week ago, I noticed that all my apps were noticeably slower over daytime. I found out that each day at 11:00, CPU usage suddenly peaks and stays very high for several hours, until late afternoon:

Looking at htop, I figured out that's a process from Mastodon:

It's apparently a tootctl task doing a prune-profiles with 2 days.
However, my cleanup cronjob starts at 23:15 and usually takes about 20-40 minutes. And the cronjob as well as all the systems in the Mastodon admin UI are set to 7 days.

So I wonder where this task is coming from and how I can re-schedule or maybe even deactivate it. Any ideas?
Right now I'm killing the task every day, but that's not how I want to spend the rest of my life.

[nebulon]

So this is the app package cleanup task for cached media https://git.cloudron.io/cloudron/mastodon-app/-/blob/master/cleanup.sh

I have recently set the default to 2 days as this is only a cache (avatar/media/cards/..) and caused a lot of small files, which slowed down backups a lot.

You can adjust this though: https://docs.cloudron.io/apps/mastodon/#cache-retention-days

Tbh I am not quite sure how the content retention settings in the UI are used, maybe those are just the defaults for the tootctl cleanup tasks? Maybe our cleanup task is not required anymore even.

[nichu42]

Ah, thanks for the clarification. I don't know why it uses so much CPU and it takes so long.

I have the following entries in my daily cronjob:

/app/code/bin/tootctl media remove --days 7
/app/code/bin/tootctl statuses remove --days 7
/app/code/bin/tootctl media remove-orphans
/app/code/bin/tootctl preview-cards remove --days 7
/app/code/bin/tootctl cache clear
/app/code/bin/tootctl accounts prune
/app/code/bin/tootctl media remove --prune-profiles --days 7

Is there a way I can disable the package cleanup task completely? I don't think I need it. The settings above did a good job for me.

[nebulon]

Are you referring to a custom cron job you have set there? As the package does contain this script https://git.cloudron.io/cloudron/mastodon-app/-/blob/master/cleanup.sh

[nichu42]

@nebulon
Yes, I have created a custom cronjob for this a few months ago when backups started to take forever due to all the small files (as you mentioned as well).
It starts at 23:15 each night.

[nebulon]

In this case you can probably remove your custom cron job and rely on the one from the package. I have to investigate which ones from your script we should add to our cleanup script then.

[nebulon]

I have added missing ones from your script to the one in the package. Only accounts prune does not seem to exist anymore.

This will then be part of the next package release.

[nichu42]

It's not in the CLI docs yet, but 'accounts prune' does work and is quite helpful.
https://github.com/mastodon/mastodon/pull/18397

If you add this too, I can indeed remove my custom cronjob. But please also add an option to control when the cleanup.sh is launched. 11:00 is not a good time for my server, I'd like to keep the cleanup processes running in the night, just before the scheduled backup.

[doodlemania2]

that's a great add, thanks for doing that @nebulon !

[nebulon]

Thanks for the hint, I have included the account pruning

[nichu42]

I've changed the setting to 7 days and the runtime of the built-in cleanup script is now down to 1 hour / day, which is absolutely fine. I'd still like to schedule it for the night, though.

[nebulon]

since it doesn't matter much I will then put the package cron from current 11 11 * * * to 11 01 * * *

[johannesjom]

This script also puts quite a lot of load on our server. Is it really still necessary to use it yourself instead of the clean-up scripts provided by Mastodon?

[girish]

@johannesjom which script are you referring to?

[nebulon]

The package cron job for cleanup does use the upstream cleanup jobs. You can also change the retention days via https://docs.cloudron.io/apps/mastodon/#cache-retention-days

[johannesjom]

@nebulon i dit asume that /admin/settings/content_retention has other scripts behind, cause there you can change the days because you can set different days for different things there.

[nichu42]

@johannesjom It seems that the internal routine does remove some content, but as most of the admin settings and options, it's very poorly documented, if at all. I have no idea which tootctl commands exactly are triggered, or when that happens.
I guess you have to go through lots of Issues and PRs to find out about this. As long as it's unclear, I think it's a good idea to have an additional cleanup script (such as the one provided by Cloudron) to make sure that the installation doesn't take too much space.

[nebulon]

at least last time I had checked the code, the UI settings were actually different from what they are now, but it is still not clear to me who and where those cleanup tasks are run or if the settings are just the default for the cli tasks we run in the package via cron.

[nichu42]

@nebulon: This seems to be a quite comprehensive and well documented clean-up script: https://codeberg.org/Fedimins/mastodon-maintenance-tasks
Maybe you want to ship it with Cloudron?

[girish]

@nichu42 good link...

I had a look at https://codeberg.org/Fedimins/mastodon-maintenance-tasks . I couldn't run the script directly because there's a bunch of hardcoded paths in it. Found some commands that maybe we can incorporate. Many of the commands are already in our cleanup script.

• tootctl accounts cull --concurrency xx - removes remote accounts that no longer exist
• tootctl accounts prune - remove accounts that have never interacted locally
• tootctl domains purge <domain> - https://docs.joinmastodon.org/admin/tootctl/#domains-purge
• run_tootctl statuses remove --days=xx - https://docs.joinmastodon.org/admin/tootctl/#statuses-remove
• tootctl media remove --days <days> --concurrency <concurrency> - https://docs.joinmastodon.org/admin/tootctl/#media-remove