Able to clean up binlog.###### files in /var/lib/mysql directory?
-
@d19dotca We can't skip the binlog because it's used for db recovery. But in the mysql addon we set
max_binlog_size = 100Mand alsoexpire_logs_days = 10. So, I will put this in the main mysql config as well.See https://dev.mysql.com/doc/refman/8.0/en/binary-log.html for importance of binlog.
edit: it's
binlog_expire_log_secondsnow - https://dev.mysql.com/worklog/task/?id=10924@girish Sounds good. I'm surprised that binlogs is needed for database recovery in a non-replicated scenario. In my day-job if we need to recover a database we just use the mysqldump created and then use the mysql command to import, nothing needed on the binlog side of things. Maybe you guys do it differently though than I'm used to.
If I edit the my.cnf file, will a future Cloudron update overwrite it? Trying to see if I should add it in now or just wait for the update so I don't mess up the update part at all.
--
Dustin Dauncey
www.d19.ca -
@girish Sounds good. I'm surprised that binlogs is needed for database recovery in a non-replicated scenario. In my day-job if we need to recover a database we just use the mysqldump created and then use the mysql command to import, nothing needed on the binlog side of things. Maybe you guys do it differently though than I'm used to.
If I edit the my.cnf file, will a future Cloudron update overwrite it? Trying to see if I should add it in now or just wait for the update so I don't mess up the update part at all.
@d19dotca Same. We also always use only mysqldump for recovery and major mysql upgrades (like the upcoming one from 5.7 to 8.0). Generally, I don't trust those direct data directory upgrade procedures
But the recovery stuff is sometimes useful when the disk goes corrupt and we want to recover quickly.From what I can make out, this issue has surfaced because expire_logs_days which was 10 is now deprecated and replaced with this log_seconds variable which has a default of 30 days...
If I edit the my.cnf file, will a future Cloudron update overwrite it? Trying to see if I should add it in now or just wait for the update so I don't mess up the update part at all.
Yes, the my.cnf will be overwritten. You can just add it it's fine, the update will merely overwrite it with the same config... Currently, I am testing with 100M max_binlog_size and 432000 (5 days) as expire_log_days.
-
Okay this seems to have worked. I took what you did in the git but made it 6 hours (21600 seconds) instead as at least a test. Restarted the mysql service and now many of the binlogs are gone.
Side question: I tried that earlier and restarted mysql but from the Cloudron Services page, and it didn't seem to work. However when I restarted mysql on the command line, it worked. Is there a difference in that process?
-
@d19dotca Same. We also always use only mysqldump for recovery and major mysql upgrades (like the upcoming one from 5.7 to 8.0). Generally, I don't trust those direct data directory upgrade procedures
But the recovery stuff is sometimes useful when the disk goes corrupt and we want to recover quickly.From what I can make out, this issue has surfaced because expire_logs_days which was 10 is now deprecated and replaced with this log_seconds variable which has a default of 30 days...
If I edit the my.cnf file, will a future Cloudron update overwrite it? Trying to see if I should add it in now or just wait for the update so I don't mess up the update part at all.
Yes, the my.cnf will be overwritten. You can just add it it's fine, the update will merely overwrite it with the same config... Currently, I am testing with 100M max_binlog_size and 432000 (5 days) as expire_log_days.
@girish said in Able to clean up binlog.###### files in /var/lib/mysql directory?:
432000 (5 days) as expire_log_days
Out of curiosity, is there a need to have 10 days of binlogs? Just curious, because I was seeing issues before the 10 days, so I'm worried that setting won't work for me (and presumably at least a few others with well-used Cloudron's on servers with limited local disk space). I guess I can overwrite it though in my instance using something in the .d directories that are included, or do those get overwritten too? If it can be a lot less (like 3 days or something) that'd be best I'd think, but maybe more time is needed for most environments and mine is just a fringe scenario given my limited disk space.
--
Dustin Dauncey
www.d19.ca -
@girish said in Able to clean up binlog.###### files in /var/lib/mysql directory?:
432000 (5 days) as expire_log_days
Out of curiosity, is there a need to have 10 days of binlogs? Just curious, because I was seeing issues before the 10 days, so I'm worried that setting won't work for me (and presumably at least a few others with well-used Cloudron's on servers with limited local disk space). I guess I can overwrite it though in my instance using something in the .d directories that are included, or do those get overwritten too? If it can be a lot less (like 3 days or something) that'd be best I'd think, but maybe more time is needed for most environments and mine is just a fringe scenario given my limited disk space.
-
@d19dotca as far as I understand the situation, it makes not too much sense to keep binlogs for db recovery for longer than the last backup (which contains the mysqldump for restore)
-
I guess setting it to something more reasonable for us like a couple of days should be plenty
@nebulon said in Able to clean up binlog.###### files in /var/lib/mysql directory?:
I guess setting it to something more reasonable for us like a couple of days should be plenty
Exactly, yes. I think it should basically be the minimal number possible / reasonable, especially when a well-used Cloudron like mine seems to generate 1-2 GB of them in just a 24 hour period. If we kept that as 10 days that'd be anywheres from 10 GB to 20 GB which is far too much disk space consumed for that, IMO. I thinking keeping them for just two days would be sufficient, heck even maybe just one day.
-
Thinking out loud here but a couple things I'm wondering about (and will try to test what I can):
- Is this issue unique to only my instance, or happening to instances of Cloudron?
- If happening to other instances, is the commonality being on Ubuntu 20.04 or MySQL 8.x?
- If others have this issue, then I'd suspect we need to enable something during Cloudron setup to handle this out of the box, or ideally just disable bin logs (if possible).
-
@d19dotca Probably I had the same problem, yesterday one instance alerted that disk was full. I didn't have time yet to check inside.
@p44 Yeah probably. I'm honestly very surprised how many people seemingly haven't run into this issue yet. I guess most using Cloudron are still on Ubuntu 18.04 rather than 20.04, or those on 20.04 have plenty of disk space where it's not a concern at all (unfortunately I don't have that luxury, haha) or they just don't use their Cloudrons nearly as much as some of us do which means they have far less writes to the database thus less binlog entries. Guess it's a "fringe" scenario for now haha but definitely still a valid one that we need to fix. I think the changes coming from Girish & Nebulon will have a great impact on that and prevent issues like I and you were running into where the disk usage goes insane from binlog files. Keeping the binlogs to just 1 or 2 days should be plenty for actively used Cloudron servers.
--
Dustin Dauncey
www.d19.ca -
All our managed instances (our old business) are on Ubuntu 20 and they don't seem to accumulate as much data. It's around 100MB everyday and so it's < 1GB for 10 days. I guess the deal is we have the luxury of extra space. What I noticed is that it seems if you start afresh on Ubuntu 20 the behavior appears different from if you migrate from Ubuntu 18. I haven't really tinkered with the combinations of upgrades.
In any case, binlog is at 2 days for both the mysql addon and the main mysql server now.
-
@p44 Yeah probably. I'm honestly very surprised how many people seemingly haven't run into this issue yet. I guess most using Cloudron are still on Ubuntu 18.04 rather than 20.04, or those on 20.04 have plenty of disk space where it's not a concern at all (unfortunately I don't have that luxury, haha) or they just don't use their Cloudrons nearly as much as some of us do which means they have far less writes to the database thus less binlog entries. Guess it's a "fringe" scenario for now haha but definitely still a valid one that we need to fix. I think the changes coming from Girish & Nebulon will have a great impact on that and prevent issues like I and you were running into where the disk usage goes insane from binlog files. Keeping the binlogs to just 1 or 2 days should be plenty for actively used Cloudron servers.
@d19dotca So reading a bit more, I think you might be right that it's of minimal use when not replicating and for replaying commands in that mode... So, I will disable it altogether. It got turned on in MySQL 8 by default but was off in MySQL 5.7.
-
@d19dotca So reading a bit more, I think you might be right that it's of minimal use when not replicating and for replaying commands in that mode... So, I will disable it altogether. It got turned on in MySQL 8 by default but was off in MySQL 5.7.
-
@p44 Yeah probably. I'm honestly very surprised how many people seemingly haven't run into this issue yet. I guess most using Cloudron are still on Ubuntu 18.04 rather than 20.04, or those on 20.04 have plenty of disk space where it's not a concern at all (unfortunately I don't have that luxury, haha) or they just don't use their Cloudrons nearly as much as some of us do which means they have far less writes to the database thus less binlog entries. Guess it's a "fringe" scenario for now haha but definitely still a valid one that we need to fix. I think the changes coming from Girish & Nebulon will have a great impact on that and prevent issues like I and you were running into where the disk usage goes insane from binlog files. Keeping the binlogs to just 1 or 2 days should be plenty for actively used Cloudron servers.
@d19dotca totally agree on this point of view.
I tryed to run:
mysql -uroot -ppassword PURGE BINARY LOGS BEFORE '2021-02-21 23:00:00';and I saved around 2,5 GB...
I think Cloudron have two missing two things:
- Control of logs amounts and cleaning function;
- Control of logs to understand potentially threat and give ability to eg. block IP or block login attempts.
-
@d19dotca totally agree on this point of view.
I tryed to run:
mysql -uroot -ppassword PURGE BINARY LOGS BEFORE '2021-02-21 23:00:00';and I saved around 2,5 GB...
I think Cloudron have two missing two things:
- Control of logs amounts and cleaning function;
- Control of logs to understand potentially threat and give ability to eg. block IP or block login attempts.
-
@p44 Agreed. It was definitely a bug. It's all fixed now though for 6.2 when it comes out.
https://git.cloudron.io/cloudron/box/-/blob/master/setup/start/mysql.cnf#L19
