Nextcloud not responding

[girish]

@tadeas is this a CIFS volume ? Moving app data to CIFS volume will not work (it doesn't support permissions and symlinks).

[Kubernetes]

@girish the Hetzner SSD Volume is mounted like a local disk - so it is not CIFS

[tadeas]

@girish it is ext4

[girish]

@tadeas Do you have anything in the logs then? If you keep the log window open and restart the app, you will get a more "complete" log.

[tadeas]

I have enabled remote SSH access.

Logs say:

box:scheduler resumeJobs: 7980df23-9257-4154-b145-c80d6cbfc579
box:tasks startTask: 17149 done. error: null
ldap undefined
ldap undefined
localstorage undefined
localstorage undefined
postgresql undefined
redis undefined
redis undefined
scheduler undefined
scheduler undefined
sendmail undefined
sendmail undefined
turn undefined
turn undefined
Jun 07 14:53:44 ==> run migration
Jun 07 14:53:45 Nextcloud is not installed - only a limited number of commands are available
Jun 07 14:53:45 Command "upgrade" is not defined.

[tadeas]

@tadeas
Healtheck error: Error: connect EHOSTUNREACH 172.18.16.8:80

[tadeas]

@tadeas I can onlu connect to the app's terminal if it is in recovery mode. Not sure it's relevant.

[Kubernetes]

I think we need the logfile when the App is NOT starting in recovery mode

[tadeas]

@Kubernetes The above is from the normal mode. Copied form the logs.

[tadeas]

@girish Based on the logs posted, do you have an idea how to go on with this?

[tadeas]

@girish when I try to get the IP of the docker container with the not responding Nextcloud app

docker exec 7380df23-9257-4154-b145-c80dbcbfc579 sh —c 'hostname —-ip-address’

I get

Error response from daemon: Container aeeeZbc4343b78fb205d9b9c55e79450694a60f 1b654272cfIc51754acabfc
3f is restarting, wait until the container is running

[tadeas]

@girish This then is the log for docker

*Jun 08 11:17:23 tadeas systemd[1]: Started Docker Application Container Engine.
Jun 08 11:17:23 tadeas dockerd[14056]: time="2023-06-08T09:17:23.776740149Z" level=info msg="API listen on /var/run/docker.sock"
Jun 08 11:17:33 tadeas dockerd[14056]: time="2023-06-08T09:17:33.284827330Z" level=info msg="ignoring event" container=fe61b92479b251c5a041d910590686c73ca50ee6e77da26e69bdf80849336f49 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jun 08 11:17:33 tadeas dockerd[14056]: time="2023-06-08T09:17:33.966635957Z" level=info msg="ignoring event" container=aeee2bc4343b78fb205d9b9c55e79450694a60f1b654272cf9c51754acabfc3f module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jun 08 11:17:41 tadeas dockerd[14056]: time="2023-06-08T09:17:41.583114294Z" level=info msg="ignoring event" container=aeee2bc4343b78fb205d9b9c55e79450694a60f1b654272cf9c51754acabfc3f module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jun 08 11:17:41 tadeas dockerd[14056]: time="2023-06-08T09:17:41.845451237Z" level=info msg="ignoring event" container=fe61b92479b251c5a041d910590686c73ca50ee6e77da26e69bdf80849336f49 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jun 08 11:17:48 tadeas dockerd[14056]: time="2023-06-08T09:17:48.857881988Z" level=info msg="ignoring event" container=aeee2bc4343b78fb205d9b9c55e79450694a60f1b654272cf9c51754acabfc3f module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jun 08 11:17:49 tadeas dockerd[14056]: time="2023-06-08T09:17:49.091000109Z" level=info msg="ignoring event" container=fe61b92479b251c5a041d910590686c73ca50ee6e77da26e69bdf80849336f49 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jun 08 11:17:54 tadeas dockerd[14056]: time="2023-06-08T09:17:54.559333319Z" level=info msg="ignoring event" container=aeee2bc4343b78fb205d9b9c55e79450694a60f1b654272cf9c51754acabfc3f module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jun 08 11:17:54 tadeas dockerd[14056]: time="2023-06-08T09:17:54.933646328Z" level=info msg="ignoring event" container=fe61b92479b251c5a041d910590686c73ca50ee6e77da26e69bdf80849336f49 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jun 08 11:17:59 tadeas dockerd[14056]: time="2023-06-08T09:17:59.949410471Z" level=info msg="ignoring event" container=aeee2bc4343b78fb205d9b9c55e79450694a60f1b654272cf9c51754acabfc3f module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jun 08 11:18:00 tadeas dockerd[14056]: time="2023-06-08T09:18:00.809874031Z" level=info msg="ignoring event" container=fe61b92479b251c5a041d910590686c73ca50ee6e77da26e69bdf80849336f49 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jun 08 11:18:07 tadeas dockerd[14056]: time="2023-06-08T09:18:07.044733887Z" level=info msg="ignoring event" container=aeee2bc4343b78fb205d9b9c55e79450694a60f1b654272cf9c51754acabfc3f module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jun 08 11:18:07 tadeas dockerd[14056]: time="2023-06-08T09:18:07.597133145Z" level=info msg="ignoring event" container=fe61b92479b251c5a041d910590686c73ca50ee6e77da26e69bdf80849336f49 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jun 08 11:18:17 tadeas dockerd[14056]: time="2023-06-08T09:18:17.137968331Z" level=info msg="ignoring event" container=aeee2bc4343b78fb205d9b9c55e79450694a60f1b654272cf9c51754acabfc3f module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jun 08 11:18:17 tadeas dockerd[14056]: time="2023-06-08T09:18:17.869097092Z" level=info msg="ignoring event" container=fe61b92479b251c5a041d910590686c73ca50ee6e77da26e69bdf80849336f49 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jun 08 11:18:33 tadeas dockerd[14056]: time="2023-06-08T09:18:33.771389179Z" level=info msg="ignoring event" container=aeee2bc4343b78fb205d9b9c55e79450694a60f1b654272cf9c51754acabfc3f module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jun 08 11:18:35 tadeas dockerd[14056]: time="2023-06-08T09:18:35.553475443Z" level=info msg="ignoring event" container=fe61b92479b251c5a041d910590686c73ca50ee6e77da26e69bdf80849336f49 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jun 08 11:19:02 tadeas dockerd[14056]: time="2023-06-08T09:19:02.825913370Z" level=info msg="ignoring event" container=aeee2bc4343b78fb205d9b9c55e79450694a60f1b654272cf9c51754acabfc3f module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jun 08 11:19:04 tadeas dockerd[14056]: time="2023-06-08T09:19:04.690648095Z" level=info msg="ignoring event" container=fe61b92479b251c5a041d910590686c73ca50ee6e77da26e69bdf80849336f49 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jun 08 11:19:39 tadeas dockerd[14056]: time="2023-06-08T09:19:39.567185708Z" level=error msg="Error setting up exec command in container aeee2bc4343b78fb205d9b9c55e79450694a60f1b654272cf9c51754acabfc3f: Container aeee2bc4343b78fb205d9b9c55e79450694a60f1b654272cf9c51754acabfc3f is restarting, wait until the container is running"
Jun 08 11:19:57 tadeas dockerd[14056]: time="2023-06-08T09:19:57.029795197Z" level=info msg="ignoring event" container=aeee2bc4343b78fb205d9b9c55e79450694a60f1b654272cf9c51754acabfc3f module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jun 08 11:19:59 tadeas dockerd[14056]: time="2023-06-08T09:19:59.168736798Z" level=info msg="ignoring event" container=fe61b92479b251c5a041d910590686c73ca50ee6e77da26e69bdf80849336f49 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jun 08 11:21:00 tadeas dockerd[14056]: time="2023-06-08T09:21:00.596425372Z" level=info msg="ignoring event" container=aeee2bc4343b78fb205d9b9c55e79450694a60f1b654272cf9c51754acabfc3f module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jun 08 11:21:02 tadeas dockerd[14056]: time="2023-06-08T09:21:02.983611725Z" level=info msg="ignoring event" container=fe61b92479b251c5a041d910590686c73ca50ee6e77da26e69bdf80849336f49 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jun 08 11:22:03 tadeas dockerd[14056]: time="2023-06-08T09:22:03.806981176Z" level=info msg="ignoring event" container=aeee2bc4343b78fb205d9b9c55e79450694a60f1b654272cf9c51754acabfc3f module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jun 08 11:22:06 tadeas dockerd[14056]: time="2023-06-08T09:22:06.401621040Z" level=info msg="ignoring event" container=fe61b92479b251c5a041d910590686c73ca50ee6e77da26e69bdf80849336f49 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jun 08 11:23:06 tadeas dockerd[14056]: time="2023-06-08T09:23:06.938708078Z" level=info msg="ignoring event" container=aeee2bc4343b78fb205d9b9c55e79450694a60f1b654272cf9c51754acabfc3f module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jun 08 11:23:09 tadeas dockerd[14056]: time="2023-06-08T09:23:09.653124335Z" level=info msg="ignoring event" container=fe61b92479b251c5a041d910590686c73ca50ee6e77da26e69bdf80849336f49 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jun 08 11:23:10 tadeas dockerd[14056]: time="2023-06-08T09:23:10.344993373Z" level=error msg="collecting stats for fe61b92479b251c5a041d910590686c73ca50ee6e77da26e69bdf80849336f49: no such container"
Jun 08 11:23:11 tadeas dockerd[14056]: time="2023-06-08T09:23:11.473463856Z" level=error msg="collecting stats for fe61b92479b251c5a041d910590686c73ca50ee6e77da26e69bdf80849336f49: no such container"
Jun 08 11:24:10 tadeas dockerd[14056]: time="2023-06-08T09:24:10.622077992Z" level=info msg="ignoring event" container=aeee2bc4343b78fb205d9b9c55e79450694a60f1b654272cf9c51754acabfc3f module=libcontainerd namespace=moby topic=/tasks/delete type="events.TaskDelete"
Jun 08 11:24:11 tadeas dockerd[14056]: time="2023-06-08T09:24:11.543530253Z" level=error msg="collecting stats for aeee2bc4343b78fb205d9b9c55e79450694a60f1b654272cf9c51754acabfc3f: no such container"
Jun 08 11:24:12 tadeas dockerd[14056]: time="2023-06-08T09:24:12.986413618Z" level=info msg="ignoring event" container=fe61b92479b251c5a041d910590686c73ca50ee6e77da26e69bdf80849336f49 module=libcontainerd namespace=moby topic=/tasks/delete type="events.TaskDelete"

[tadeas]

When I put Nextcloud into repair mode, open the terminal, then and then do:

occ status

it says:

Nextcloud is not installed - only a limited number of commands are available

• installed: false
• version: 26.0.2.1
• versionstring: 26.0.2
• edition:
• maintenance: false
• needsDbUpgrade: false
• productname: Nextcloud
• extendedSupport: false

[tadeas]

@girish per https://docs.cloudron.io/apps/nextcloud/#fixing-a-broken-install I have set debug to true in the config

in the logs, this does not produce anything new

the log only cycles on: Healtheck error: Error: connect ECONNREFUSED 172.18.16.8:80

should I abandon the install and somehow extract the data?

[tadeas]

I got the debug working. There was a problem with a volume that that was made available to nextcloud (but nextcloud doesn't depend on it for functionng, it is just data repository). I have removed the volumes, now that erro is gone.

But the problem persists.

In normal mode, Nextcloud doesn't start:

Jun 09 12:43:40 => Healtheck error: Error: connect ECONNREFUSED 172.18.16.8:80
Jun 09 12:43:40 ==> run migration
Jun 09 12:43:40 Nextcloud is not installed - only a limited number of commands are available

In recovery mode, it only produces this:

Jun 09 12:44:40 => Healtheck error: Error: connect ECONNREFUSED 172.18.16.8:80

[girish]

@tadeas it seems permissions don't work on the volume . It's also mounted by via sshfs/fuse. I am not sure how well this works. Any reason not to use NFS if it is block storage?

# chown root /mnt/storage_box/xx
chown: changing ownership of 'xx': Permission denied

[girish]

Turns out while that storage volume has the issue it was not really used.

The issue was that /app/data/config/config.php has the data directory set to /app/code/data instead of /app/data . Not sure how this happened. But resetting that config value makes it come up again.

[tadeas]

@girish I think this was a working solution two years ago and I have not touched it since then. It was only used to mount a directory so that it would be visible form Nextcloud. While looking for a solution I have removed it from nextcloud so that's why it was not used.

[tadeas]

@girish Thanks a lot, i did not notice that.