Issues after full disk

[SolarSimon]

Hi there, my Nextcloud caused some trouble, when my server disc was full, cause of docker-volumes.
Excactly this issue https://forum.cloudron.io/topic/11112/how-to-free-up-disk-space-docker-volumes/3

My Server caused lots of trouble. The DNS unbound stopped. I could not mount my sshfs volumes. After hours I tried creating a new mount, suddenly I could mount the existing volumes and backup storage.

It was still not possible to log in to my Nextcloud. I figured out, config.php was corrupt. Some lines where simply missing. I restored from an backup and could log in again.

But still the Server has some issues. Nextcloud Notes is thowing errors (Saving note has failed HTTP 500 RuntumeException), Collective refuses to open pages (could not fetch pages). But I don't see log entries.

I have now spent hours trying to get the server up and running again.
The fact that the config.php was faulty makes me suspicious, maybe other files are also faulty.
Now I can't get any further and need help.

I did all the troubleshooting in the cloudron docs, scanned Nextcloud and Box logs without finding something.

Looks all fine, except journalctl -xe:

pam_unix(sshd:auth): check pass; user unknown
pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=191.243.149.210
Failed password for invalid user admin from 191.243.149.210 port 33254 ssh2
pam_unix(sshd:auth): check pass; user unknown
Packet dropped: IN=eth0 OUT= MAC=96:00:01:07:41:fe:d2:74:7f:6e:37:e3:08:00 SRC=104.248.135.66 DST=116.202.17.21 LEN=60 TOS=0x00 PREC=0x00 TTL=243 ID=4003 PROTO=T>
Failed password for invalid user admin from 191.243.149.210 port 33254 ssh2
error: Received disconnect from 191.243.149.210 port 33254:3: com.jcraft.jsch.JSchException: Auth cancel [preauth]
Disconnected from invalid user admin 191.243.149.210 port 33254 [preauth]
PAM 1 more authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=191.243.149.210

Most things are doing fine now, files are accessible, synchronization works, cal-dav works.
What I see currently is collective and notes.
My guess is, that appsdata could be corrupt or permission issue?
Some Ideas where to go?

[nebulon]

If the server runs out of space many things may be shaky. Have you worked through https://docs.cloudron.io/troubleshooting/#recovery-after-disk-full already?

For Nextclou specifically, it keeps trying to write .php files onto disk, which get corrupted if there is no disk space available

The logs you posted only show failed auth attempts mostly from bots, unfortunately that will always happen but is harmless if your SSH is using only key pairs.

[SolarSimon]

@nebulon said in Issues after full disk:

Have you worked through https://docs.cloudron.io/troubleshooting/#recovery-after-disk-full already?

yep, what I found was DNS stopped and the log from journalctl.

Do I have to recover MySQL?

@nebulon said in Issues after full disk:

For Nextclou specifically, it keeps trying to write .php files onto disk, which get corrupted if there is no disk space available

I restored Nextcloud App from a backup. Everything should be fine then, if it was before disk was full? Or some config to be done?

@nebulon said in Issues after full disk:

If the server runs out of space many things may be shaky.

Why did it even happen? Why do I not get quota warning?
Now I have to deal with a corrupted installation

[girish]

@SolarSimon said in Issues after full disk:

Why did it even happen? Why do I not get quota warning?

On linux, there is no way for this unfortunately at file system level (with ext4 anyway). There is user and group quota (https://wiki.archlinux.org/title/Disk_quota) but we have to investigate if we can make this work with containers reliably.