Nextcloud not responding

[tadeas]

Two of my nextcloud instances are not responding.

For both, this is in the logs:

Nextcloud is not installed - only a limited number of commands are available

and

Healtheck error: Error: connect EHOSTUNREACH 172.18.16.8:80

Reddis is running.

I have been unable to start up nextcloud after moving the apps to a new volume due to low space.

Thanks

[Kubernetes]

How did you connect the new volume? Is it physical or a network volume?

[tadeas]

@Kubernetes It's Hetzner SSD Volume, it's a mountpoint mounted via Cloudron's web interface (Volumes -> Add Volume). Then the app was moved to the volume via Cloudron's web interface as well (a different app was OK with the move).

It might be an ongoing issue from the time of low/no space on the disk.

[Kubernetes]

Hm, I did exactly the same just a few days ago. My NextCloud continued to work after the move to the SSD. Is there nothing more in the Logs?

[girish]

@tadeas is this a CIFS volume ? Moving app data to CIFS volume will not work (it doesn't support permissions and symlinks).

[Kubernetes]

@girish the Hetzner SSD Volume is mounted like a local disk - so it is not CIFS

[tadeas]

@girish it is ext4

[girish]

@tadeas Do you have anything in the logs then? If you keep the log window open and restart the app, you will get a more "complete" log.

[tadeas]

I have enabled remote SSH access.

Logs say:

box:scheduler resumeJobs: 7980df23-9257-4154-b145-c80d6cbfc579
box:tasks startTask: 17149 done. error: null
ldap undefined
ldap undefined
localstorage undefined
localstorage undefined
postgresql undefined
redis undefined
redis undefined
scheduler undefined
scheduler undefined
sendmail undefined
sendmail undefined
turn undefined
turn undefined
Jun 07 14:53:44 ==> run migration
Jun 07 14:53:45 Nextcloud is not installed - only a limited number of commands are available
Jun 07 14:53:45 Command "upgrade" is not defined.

[tadeas]

@tadeas
Healtheck error: Error: connect EHOSTUNREACH 172.18.16.8:80

[tadeas]

@tadeas I can onlu connect to the app's terminal if it is in recovery mode. Not sure it's relevant.

[Kubernetes]

I think we need the logfile when the App is NOT starting in recovery mode

[tadeas]

@Kubernetes The above is from the normal mode. Copied form the logs.

[tadeas]

@girish Based on the logs posted, do you have an idea how to go on with this?

[tadeas]

@girish when I try to get the IP of the docker container with the not responding Nextcloud app

docker exec 7380df23-9257-4154-b145-c80dbcbfc579 sh —c 'hostname —-ip-address’

I get

Error response from daemon: Container aeeeZbc4343b78fb205d9b9c55e79450694a60f 1b654272cfIc51754acabfc
3f is restarting, wait until the container is running

[tadeas]

@girish This then is the log for docker

*Jun 08 11:17:23 tadeas systemd[1]: Started Docker Application Container Engine.
Jun 08 11:17:23 tadeas dockerd[14056]: time="2023-06-08T09:17:23.776740149Z" level=info msg="API listen on /var/run/docker.sock"
Jun 08 11:17:33 tadeas dockerd[14056]: time="2023-06-08T09:17:33.284827330Z" level=info msg="ignoring event" container=fe61b92479b251c5a041d910590686c73ca50ee6e77da26e69bdf80849336f49 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jun 08 11:17:33 tadeas dockerd[14056]: time="2023-06-08T09:17:33.966635957Z" level=info msg="ignoring event" container=aeee2bc4343b78fb205d9b9c55e79450694a60f1b654272cf9c51754acabfc3f module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jun 08 11:17:41 tadeas dockerd[14056]: time="2023-06-08T09:17:41.583114294Z" level=info msg="ignoring event" container=aeee2bc4343b78fb205d9b9c55e79450694a60f1b654272cf9c51754acabfc3f module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jun 08 11:17:41 tadeas dockerd[14056]: time="2023-06-08T09:17:41.845451237Z" level=info msg="ignoring event" container=fe61b92479b251c5a041d910590686c73ca50ee6e77da26e69bdf80849336f49 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jun 08 11:17:48 tadeas dockerd[14056]: time="2023-06-08T09:17:48.857881988Z" level=info msg="ignoring event" container=aeee2bc4343b78fb205d9b9c55e79450694a60f1b654272cf9c51754acabfc3f module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jun 08 11:17:49 tadeas dockerd[14056]: time="2023-06-08T09:17:49.091000109Z" level=info msg="ignoring event" container=fe61b92479b251c5a041d910590686c73ca50ee6e77da26e69bdf80849336f49 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jun 08 11:17:54 tadeas dockerd[14056]: time="2023-06-08T09:17:54.559333319Z" level=info msg="ignoring event" container=aeee2bc4343b78fb205d9b9c55e79450694a60f1b654272cf9c51754acabfc3f module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jun 08 11:17:54 tadeas dockerd[14056]: time="2023-06-08T09:17:54.933646328Z" level=info msg="ignoring event" container=fe61b92479b251c5a041d910590686c73ca50ee6e77da26e69bdf80849336f49 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jun 08 11:17:59 tadeas dockerd[14056]: time="2023-06-08T09:17:59.949410471Z" level=info msg="ignoring event" container=aeee2bc4343b78fb205d9b9c55e79450694a60f1b654272cf9c51754acabfc3f module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jun 08 11:18:00 tadeas dockerd[14056]: time="2023-06-08T09:18:00.809874031Z" level=info msg="ignoring event" container=fe61b92479b251c5a041d910590686c73ca50ee6e77da26e69bdf80849336f49 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jun 08 11:18:07 tadeas dockerd[14056]: time="2023-06-08T09:18:07.044733887Z" level=info msg="ignoring event" container=aeee2bc4343b78fb205d9b9c55e79450694a60f1b654272cf9c51754acabfc3f module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jun 08 11:18:07 tadeas dockerd[14056]: time="2023-06-08T09:18:07.597133145Z" level=info msg="ignoring event" container=fe61b92479b251c5a041d910590686c73ca50ee6e77da26e69bdf80849336f49 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jun 08 11:18:17 tadeas dockerd[14056]: time="2023-06-08T09:18:17.137968331Z" level=info msg="ignoring event" container=aeee2bc4343b78fb205d9b9c55e79450694a60f1b654272cf9c51754acabfc3f module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jun 08 11:18:17 tadeas dockerd[14056]: time="2023-06-08T09:18:17.869097092Z" level=info msg="ignoring event" container=fe61b92479b251c5a041d910590686c73ca50ee6e77da26e69bdf80849336f49 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jun 08 11:18:33 tadeas dockerd[14056]: time="2023-06-08T09:18:33.771389179Z" level=info msg="ignoring event" container=aeee2bc4343b78fb205d9b9c55e79450694a60f1b654272cf9c51754acabfc3f module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jun 08 11:18:35 tadeas dockerd[14056]: time="2023-06-08T09:18:35.553475443Z" level=info msg="ignoring event" container=fe61b92479b251c5a041d910590686c73ca50ee6e77da26e69bdf80849336f49 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jun 08 11:19:02 tadeas dockerd[14056]: time="2023-06-08T09:19:02.825913370Z" level=info msg="ignoring event" container=aeee2bc4343b78fb205d9b9c55e79450694a60f1b654272cf9c51754acabfc3f module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jun 08 11:19:04 tadeas dockerd[14056]: time="2023-06-08T09:19:04.690648095Z" level=info msg="ignoring event" container=fe61b92479b251c5a041d910590686c73ca50ee6e77da26e69bdf80849336f49 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jun 08 11:19:39 tadeas dockerd[14056]: time="2023-06-08T09:19:39.567185708Z" level=error msg="Error setting up exec command in container aeee2bc4343b78fb205d9b9c55e79450694a60f1b654272cf9c51754acabfc3f: Container aeee2bc4343b78fb205d9b9c55e79450694a60f1b654272cf9c51754acabfc3f is restarting, wait until the container is running"
Jun 08 11:19:57 tadeas dockerd[14056]: time="2023-06-08T09:19:57.029795197Z" level=info msg="ignoring event" container=aeee2bc4343b78fb205d9b9c55e79450694a60f1b654272cf9c51754acabfc3f module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jun 08 11:19:59 tadeas dockerd[14056]: time="2023-06-08T09:19:59.168736798Z" level=info msg="ignoring event" container=fe61b92479b251c5a041d910590686c73ca50ee6e77da26e69bdf80849336f49 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jun 08 11:21:00 tadeas dockerd[14056]: time="2023-06-08T09:21:00.596425372Z" level=info msg="ignoring event" container=aeee2bc4343b78fb205d9b9c55e79450694a60f1b654272cf9c51754acabfc3f module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jun 08 11:21:02 tadeas dockerd[14056]: time="2023-06-08T09:21:02.983611725Z" level=info msg="ignoring event" container=fe61b92479b251c5a041d910590686c73ca50ee6e77da26e69bdf80849336f49 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jun 08 11:22:03 tadeas dockerd[14056]: time="2023-06-08T09:22:03.806981176Z" level=info msg="ignoring event" container=aeee2bc4343b78fb205d9b9c55e79450694a60f1b654272cf9c51754acabfc3f module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jun 08 11:22:06 tadeas dockerd[14056]: time="2023-06-08T09:22:06.401621040Z" level=info msg="ignoring event" container=fe61b92479b251c5a041d910590686c73ca50ee6e77da26e69bdf80849336f49 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jun 08 11:23:06 tadeas dockerd[14056]: time="2023-06-08T09:23:06.938708078Z" level=info msg="ignoring event" container=aeee2bc4343b78fb205d9b9c55e79450694a60f1b654272cf9c51754acabfc3f module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jun 08 11:23:09 tadeas dockerd[14056]: time="2023-06-08T09:23:09.653124335Z" level=info msg="ignoring event" container=fe61b92479b251c5a041d910590686c73ca50ee6e77da26e69bdf80849336f49 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jun 08 11:23:10 tadeas dockerd[14056]: time="2023-06-08T09:23:10.344993373Z" level=error msg="collecting stats for fe61b92479b251c5a041d910590686c73ca50ee6e77da26e69bdf80849336f49: no such container"
Jun 08 11:23:11 tadeas dockerd[14056]: time="2023-06-08T09:23:11.473463856Z" level=error msg="collecting stats for fe61b92479b251c5a041d910590686c73ca50ee6e77da26e69bdf80849336f49: no such container"
Jun 08 11:24:10 tadeas dockerd[14056]: time="2023-06-08T09:24:10.622077992Z" level=info msg="ignoring event" container=aeee2bc4343b78fb205d9b9c55e79450694a60f1b654272cf9c51754acabfc3f module=libcontainerd namespace=moby topic=/tasks/delete type="events.TaskDelete"
Jun 08 11:24:11 tadeas dockerd[14056]: time="2023-06-08T09:24:11.543530253Z" level=error msg="collecting stats for aeee2bc4343b78fb205d9b9c55e79450694a60f1b654272cf9c51754acabfc3f: no such container"
Jun 08 11:24:12 tadeas dockerd[14056]: time="2023-06-08T09:24:12.986413618Z" level=info msg="ignoring event" container=fe61b92479b251c5a041d910590686c73ca50ee6e77da26e69bdf80849336f49 module=libcontainerd namespace=moby topic=/tasks/delete type="events.TaskDelete"

[tadeas]

When I put Nextcloud into repair mode, open the terminal, then and then do:

occ status

it says:

Nextcloud is not installed - only a limited number of commands are available

• installed: false
• version: 26.0.2.1
• versionstring: 26.0.2
• edition:
• maintenance: false
• needsDbUpgrade: false
• productname: Nextcloud
• extendedSupport: false

[tadeas]

@girish per https://docs.cloudron.io/apps/nextcloud/#fixing-a-broken-install I have set debug to true in the config

in the logs, this does not produce anything new

the log only cycles on: Healtheck error: Error: connect ECONNREFUSED 172.18.16.8:80

should I abandon the install and somehow extract the data?

[tadeas]

I got the debug working. There was a problem with a volume that that was made available to nextcloud (but nextcloud doesn't depend on it for functionng, it is just data repository). I have removed the volumes, now that erro is gone.

But the problem persists.

In normal mode, Nextcloud doesn't start:

Jun 09 12:43:40 => Healtheck error: Error: connect ECONNREFUSED 172.18.16.8:80
Jun 09 12:43:40 ==> run migration
Jun 09 12:43:40 Nextcloud is not installed - only a limited number of commands are available

In recovery mode, it only produces this:

Jun 09 12:44:40 => Healtheck error: Error: connect ECONNREFUSED 172.18.16.8:80

[girish]

@tadeas it seems permissions don't work on the volume . It's also mounted by via sshfs/fuse. I am not sure how well this works. Any reason not to use NFS if it is block storage?

# chown root /mnt/storage_box/xx
chown: changing ownership of 'xx': Permission denied