SOLVED Backups failing due to lack of disk space, "Everything else" category consumes 50%

[infogulch]

My backups have been failing recently because of disk space. This wouldn't be surprising (if I use the space I use the space...) except over 50% of the disk space is in the "everything else" category:

I ran some du commands locally and found that mysql is using 22GB by itself, mostly in 100M binlog.NNNNNN files:

root@localhost:/var/lib/mysql# du -sh *
4.0K    auto.cnf
4.0K    binlog.000001
4.0K    binlog.000002
120K    binlog.000003
101M    binlog.000004
101M    binlog.000005
101M    binlog.000006
101M    binlog.000007
101M    binlog.000008
59M     binlog.000009
101M    binlog.000010
101M    binlog.000011
101M    binlog.000012
101M    binlog.000013
101M    binlog.000014
71M     binlog.000015
101M    binlog.000016
101M    binlog.000017
101M    binlog.000018
101M    binlog.000019
101M    binlog.000020
101M    binlog.000021
19M     binlog.000022
101M    binlog.000023
101M    binlog.000024
101M    binlog.000025
101M    binlog.000026
101M    binlog.000027
101M    binlog.000028
19M     binlog.000029
101M    binlog.000030
101M    binlog.000031
101M    binlog.000032
101M    binlog.000033
101M    binlog.000034
101M    binlog.000035
85M     binlog.000036
101M    binlog.000037
101M    binlog.000038
101M    binlog.000039
101M    binlog.000040
101M    binlog.000041
101M    binlog.000042
93M     binlog.000043
101M    binlog.000044
101M    binlog.000045
101M    binlog.000046
101M    binlog.000047
101M    binlog.000048
101M    binlog.000049
88M     binlog.000050
101M    binlog.000051
101M    binlog.000052
101M    binlog.000053
101M    binlog.000054
101M    binlog.000055
101M    binlog.000056
90M     binlog.000057
101M    binlog.000058
101M    binlog.000059
101M    binlog.000060
101M    binlog.000061
101M    binlog.000062
101M    binlog.000063
91M     binlog.000064
101M    binlog.000065
101M    binlog.000066
101M    binlog.000067
101M    binlog.000068
101M    binlog.000069
101M    binlog.000070
68M     binlog.000071
101M    binlog.000072
101M    binlog.000073
101M    binlog.000074
101M    binlog.000075
101M    binlog.000076
101M    binlog.000077
69M     binlog.000078
101M    binlog.000079
101M    binlog.000080
101M    binlog.000081
101M    binlog.000082
101M    binlog.000083
101M    binlog.000084
71M     binlog.000085
101M    binlog.000086
101M    binlog.000087
101M    binlog.000088
101M    binlog.000089
101M    binlog.000090
101M    binlog.000091
70M     binlog.000092
101M    binlog.000093
101M    binlog.000094
45M     binlog.000095
101M    binlog.000096
101M    binlog.000097
101M    binlog.000098
101M    binlog.000099
32M     binlog.000100
101M    binlog.000101
101M    binlog.000102
101M    binlog.000103
101M    binlog.000104
101M    binlog.000105
101M    binlog.000106
101M    binlog.000107
101M    binlog.000108
51M     binlog.000109
101M    binlog.000110
101M    binlog.000111
101M    binlog.000112
101M    binlog.000113
101M    binlog.000114
101M    binlog.000115
101M    binlog.000116
101M    binlog.000117
101M    binlog.000118
101M    binlog.000119
101M    binlog.000120
101M    binlog.000121
101M    binlog.000122
101M    binlog.000123
101M    binlog.000124
101M    binlog.000125
101M    binlog.000126
36M     binlog.000127
101M    binlog.000128
101M    binlog.000129
101M    binlog.000130
101M    binlog.000131
101M    binlog.000132
101M    binlog.000133
101M    binlog.000134
101M    binlog.000135
101M    binlog.000136
48M     binlog.000137
101M    binlog.000138
101M    binlog.000139
101M    binlog.000140
101M    binlog.000141
101M    binlog.000142
101M    binlog.000143
101M    binlog.000144
101M    binlog.000145
101M    binlog.000146
101M    binlog.000147
101M    binlog.000148
101M    binlog.000149
15M     binlog.000150
101M    binlog.000151
101M    binlog.000152
101M    binlog.000153
101M    binlog.000154
101M    binlog.000155
101M    binlog.000156
101M    binlog.000157
101M    binlog.000158
101M    binlog.000159
101M    binlog.000160
101M    binlog.000161
101M    binlog.000162
100M    binlog.000163
101M    binlog.000164
101M    binlog.000165
101M    binlog.000166
101M    binlog.000167
101M    binlog.000168
101M    binlog.000169
101M    binlog.000170
101M    binlog.000171
101M    binlog.000172
101M    binlog.000173
9.7M    binlog.000174
101M    binlog.000175
101M    binlog.000176
101M    binlog.000177
101M    binlog.000178
101M    binlog.000179
101M    binlog.000180
101M    binlog.000181
101M    binlog.000182
101M    binlog.000183
101M    binlog.000184
96M     binlog.000185
101M    binlog.000186
101M    binlog.000187
101M    binlog.000188
101M    binlog.000189
101M    binlog.000190
101M    binlog.000191
101M    binlog.000192
101M    binlog.000193
101M    binlog.000194
101M    binlog.000195
95M     binlog.000196
101M    binlog.000197
101M    binlog.000198
101M    binlog.000199
101M    binlog.000200
101M    binlog.000201
101M    binlog.000202
101M    binlog.000203
101M    binlog.000204
101M    binlog.000205
101M    binlog.000206
101M    binlog.000207
33M     binlog.000208
101M    binlog.000209
101M    binlog.000210
101M    binlog.000211
101M    binlog.000212
101M    binlog.000213
101M    binlog.000214
101M    binlog.000215
101M    binlog.000216
101M    binlog.000217
101M    binlog.000218
28M     binlog.000219
101M    binlog.000220
101M    binlog.000221
101M    binlog.000222
101M    binlog.000223
101M    binlog.000224
101M    binlog.000225
101M    binlog.000226
101M    binlog.000227
80M     binlog.000228
38M     binlog.000229
101M    binlog.000230
101M    binlog.000231
101M    binlog.000232
101M    binlog.000233
101M    binlog.000234
101M    binlog.000235
41M     binlog.000236
101M    binlog.000237
101M    binlog.000238
101M    binlog.000239
101M    binlog.000240
101M    binlog.000241
13M     binlog.000242
4.0K    binlog.index
45M     box
4.0K    ca-key.pem
4.0K    ca.pem
4.0K    client-cert.pem
4.0K    client-key.pem
0       debian-5.7.flag
192K    #ib_16384_0.dblwr
8.2M    #ib_16384_1.dblwr
8.0K    ib_buffer_pool
12M     ibdata1
48M     ib_logfile0
48M     ib_logfile1
12M     ibtmp1
804K    #innodb_temp

It looks like regular maintenance by running PURGE BINARY LOGS BEFORE '<some date>'; is expected. Does cloudron do this today?

This isn't an emergency so no rush (Happy Thanksgiving!). Thanks!

[nebulon]

@infogulch binlog is actually disabled since a few releases exactly because of this and also binlogs serve no real purpose the way Cloudron uses that mysql instance. Either way you can safely purge those, but can you first check how old they are? Also check, if /etc/mysql/mysql.cnf contains skip-log-bin.

[nebulon]

@infogulch binlog is actually disabled since a few releases exactly because of this and also binlogs serve no real purpose the way Cloudron uses that mysql instance. Either way you can safely purge those, but can you first check how old they are? Also check, if /etc/mysql/mysql.cnf contains skip-log-bin.

[infogulch]

@nebulon Good question, and thanks for the context. It seems they were all created in feb/march this year. Maybe it would have been a good idea to purge the binlogs as part of that release.

After logging in with mysql -uroot -ppassword as described in troubleshooting, PURGE BINARY LOGS didn't do anything and SHOW BINARY LOGS just displayed an error that binlogs were not enabled, so I assume that mysql doesn't recognize them anymore, especially considering their age. So I ran rm binlog.*; I hope that didn't fuck it ¯_ (ツ)_/¯. The space is back now at least.

[necrevistonnezr]

@infogulch thanks for this topic, I had GB of binlogs as well!