After disk full incident, can't restart Nextcloud App
-
Nextcloud became unresponsive when the disk became full. I've cleared out 25G worth of free space. I restarted the server and restarted the app but Nextcloud is still unresponsive. All other apps on this Cloudron are working.
In the logs I'm seeing this:
This is what the log says when I try to start the Nextcloud app:Jul 16 14:10:43 box:apphealthmonitor app health: 15 running / 1 stopped / 1 unresponsive Jul 16 14:11:16 box:locker Acquired : apptask Jul 16 14:11:16 box:scheduler suspendJobs: 0eb292a7-d57c-42bc-9906-cbb13eebbcbd Jul 16 14:11:16 box:tasks startTask - starting task 6400 with options {"timeout":72000000,"nice":15,"memoryLimit":400,"logFile":"/home/yellowtent/platformdata/logs/0eb292a7-d57c-42bc-9906-cbb13eebbcbd/apptask.log"}. logs at /home/yellowtent/platformdata/logs/0eb292a7-d57c-42bc-9906-cbb13eebbcbd/apptask.log Jul 16 14:11:16 box:shell startTask spawn: /usr/bin/sudo -S -E /home/yellowtent/box/src/scripts/starttask.sh 6400 /home/yellowtent/platformdata/logs/0eb292a7-d57c-42bc-9906-cbb13eebbcbd/apptask.log 15 400 Jul 16 14:11:16 box:shell startTask (stderr): Running as unit: box-task-6400.service Jul 16 14:11:17 box:shell startTask (stderr): Finished with result: success processes terminated with: code=exited/status=0 runtime: 1.368s Jul 16 14:11:17 box:shell startTask (stdout): Service box-task-6400 finished with exit code 0 Jul 16 14:11:17 box:tasks startTask: 6400 completed with code 0 Jul 16 14:11:17 box:apps scheduleTask: task 6400 of 0eb292a7-d57c-42bc-9906-cbb13eebbcbd completed Jul 16 14:11:17 box:locker Released : apptask Jul 16 14:11:17 box:scheduler resumeJobs: 0eb292a7-d57c-42bc-9906-cbb13eebbcbd Jul 16 14:11:17 box:tasks startTask: 6400 done. error: null Jul 16 14:11:20 box:apphealthmonitor app health: 15 running / 2 stopped / 0 unresponsive Jul 16 14:11:27 box:locker Acquired : apptask Jul 16 14:11:27 box:scheduler suspendJobs: 0eb292a7-d57c-42bc-9906-cbb13eebbcbd Jul 16 14:11:27 box:tasks startTask - starting task 6401 with options {"timeout":72000000,"nice":15,"memoryLimit":400,"logFile":"/home/yellowtent/platformdata/logs/0eb292a7-d57c-42bc-9906-cbb13eebbcbd/apptask.log"}. logs at /home/yellowtent/platformdata/logs/0eb292a7-d57c-42bc-9906-cbb13eebbcbd/apptask.log Jul 16 14:11:27 box:shell startTask spawn: /usr/bin/sudo -S -E /home/yellowtent/box/src/scripts/starttask.sh 6401 /home/yellowtent/platformdata/logs/0eb292a7-d57c-42bc-9906-cbb13eebbcbd/apptask.log 15 400 Jul 16 14:11:27 box:shell startTask (stderr): Running as unit: box-task-6401.service Jul 16 14:11:28 box:apphealthmonitor app health: 15 running / 1 stopped / 1 unresponsive Jul 16 14:11:29 box:shell startTask (stderr): Finished with result: success processes terminated with: code=exited/status=0 runtime: 1.634s Jul 16 14:11:29 box:shell startTask (stdout): Service box-task-6401 finished with exit code 0 Jul 16 14:11:29 box:tasks startTask: 6401 completed with code 0 Jul 16 14:11:29 box:apps scheduleTask: task 6401 of 0eb292a7-d57c-42bc-9906-cbb13eebbcbd completed Jul 16 14:11:29 box:locker Released : apptask Jul 16 14:11:29 box:scheduler resumeJobs: 0eb292a7-d57c-42bc-9906-cbb13eebbcbd Jul 16 14:11:29 box:tasks startTask: 6401 done. error: null``` Thanks so much for your help.
-
@Shai the logs you posted are the box logs. Can you check the app logs instead? https://docs.cloudron.io/apps/#log-viewer
-
-
Here is the log that occured after I clicked "start app" button:
Jul 17 08:58:04 box:settings initCache: pre-load settings Jul 17 08:58:04 box:taskworker Starting task 6420. Logs are at /home/yellowtent/platformdata/logs/0eb292a7-d57c-42bc-9906-cbb13eebbcbd/apptask.log Jul 17 08:58:04 box:apptask run: startTask installationState: pending_start runState: running Jul 17 08:58:04 box:tasks update 6420: {"percent":10,"message":"Starting app services"} Jul 17 08:58:05 2023-07-17 12:58:05,457 CRIT Supervisor is running as root. Privileges were not dropped because no user is specified in the config file. If you intend to run as root, you can set user=root in the config file to avoid this message. Jul 17 08:58:05 2023-07-17 12:58:05,457 INFO Included extra file "/etc/supervisor/conf.d/redis-service.conf" during parsing Jul 17 08:58:05 2023-07-17 12:58:05,457 INFO Included extra file "/etc/supervisor/conf.d/redis.conf" during parsing Jul 17 08:58:05 2023-07-17 12:58:05,466 CRIT Server 'inet_http_server' running without any HTTP authentication checking Jul 17 08:58:05 2023-07-17 12:58:05,466 INFO RPC interface 'supervisor' initialized Jul 17 08:58:05 2023-07-17 12:58:05,467 CRIT Server 'unix_http_server' running without any HTTP authentication checking Jul 17 08:58:05 2023-07-17 12:58:05,467 INFO RPC interface 'supervisor' initialized Jul 17 08:58:05 2023-07-17 12:58:05,468 INFO supervisord started with pid 1 Jul 17 08:58:05 Starting supervisor Jul 17 08:58:05 box:tasks update 6420: {"percent":35,"message":"Starting container"} Jul 17 08:58:05 box:tasks update 6420: {"percent":80,"message":"Configuring reverse proxy"} Jul 17 08:58:05 box:reverseproxy providerMatchesSync: subject=CN = *.flipcloud.net domain=*.flipcloud.net issuer=C = US, O = Let's Encrypt, CN = R3 wildcard=true/true prod=true/true issuerMismatch=false wildcardMismatch=false match=true Jul 17 08:58:05 box:reverseproxy ensureCertificate: shai.flipcloud.net acme cert exists and is up to date Jul 17 08:58:05 box:reverseproxy expiryDate: subject=CN = *.flipcloud.net notBefore=Jun 12 17:11:05 2023 GMT notAfter=Sep 10 17:11:04 2023 GMT daysLeft=55.175674791666665 Jul 17 08:58:05 box:reverseproxy needsRenewal: false. force: false Jul 17 08:58:05 box:reverseproxy writeAppLocationNginxConfig: writing config for "shai.flipcloud.net" to /home/yellowtent/platformdata/nginx/applications/0eb292a7-d57c-42bc-9906-cbb13eebbcbd/shai.flipcloud.net.conf with options {"sourceDir":"/home/yellowtent/box","vhost":"shai.flipcloud.net","hasIPv6":true,"ip":"172.18.17.145","port":80,"endpoint":"app","redirectTo":null,"certFilePath":"/home/yellowtent/platformdata/nginx/cert/_.flipcloud.net.cert","keyFilePath":"/home/yellowtent/platformdata/nginx/cert/_.flipcloud.net.key","robotsTxtQuoted":null,"cspQuoted":null,"hideHeaders":[],"proxyAuth":{"enabled":false,"id":"0eb292a7-d57c-42bc-9906-cbb13eebbcbd","location":"/"},"upstreamUri":"","ocsp":true,"hstsPreload":false} Jul 17 08:58:05 box:shell reload spawn: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/restartservice.sh nginx Jul 17 08:58:06 12:C 17 Jul 2023 12:58:06.488 # Configuration loaded Jul 17 08:58:06 12:C 17 Jul 2023 12:58:06.488 # Redis version=6.0.16, bits=64, commit=00000000, modified=0, pid=12, just started Jul 17 08:58:06 12:C 17 Jul 2023 12:58:06.488 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo Jul 17 08:58:06 12:M 17 Jul 2023 12:58:06.489 # Server initialized Jul 17 08:58:06 12:M 17 Jul 2023 12:58:06.489 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect. Jul 17 08:58:06 12:M 17 Jul 2023 12:58:06.489 * Running mode=standalone, port=6379. Jul 17 08:58:06 12:M 17 Jul 2023 12:58:06.490 * Loading RDB produced by version 6.0.16 Jul 17 08:58:06 12:M 17 Jul 2023 12:58:06.490 * RDB age 36230 seconds Jul 17 08:58:06 12:M 17 Jul 2023 12:58:06.490 * RDB memory usage when created 9.32 Mb Jul 17 08:58:06 12:M 17 Jul 2023 12:58:06.536 * DB loaded from disk: 0.046 seconds Jul 17 08:58:06 12:M 17 Jul 2023 12:58:06.537 * Ready to accept connections Jul 17 08:58:06 2023-07-17 12:58:06,472 INFO spawned: 'redis' with pid 12 Jul 17 08:58:06 2023-07-17 12:58:06,477 INFO spawned: 'redis-service' with pid 13 Jul 17 08:58:06 Redis service endpoint listening on http://:::3000 Jul 17 08:58:06 box:tasks update 6420: {"percent":100,"message":"Done"} Jul 17 08:58:06 box:tasks setCompleted - 6420: {"result":null,"error":null} Jul 17 08:58:06 box:tasks update 6420: {"percent":100,"result":null,"error":null} Jul 17 08:58:06 box:taskworker Task took 1.49 seconds Jul 17 08:58:07 2023-07-17 12:58:07,701 INFO success: redis entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) Jul 17 08:58:07 2023-07-17 12:58:07,701 INFO success: redis-service entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) Jul 17 08:59:21 ==> Update app: activity Jul 17 08:59:21 ==> Update app: admin_audit Jul 17 08:59:21 ==> Update app: bruteforcesettings Jul 17 08:59:21 ==> Update app: circles Jul 17 08:59:21 ==> Update app: cloud_federation_api Jul 17 08:59:21 ==> Update app: comments Jul 17 08:59:21 ==> Update app: contactsinteraction Jul 17 08:59:21 ==> Update app: dashboard Jul 17 08:59:21 ==> Update app: dav Jul 17 08:59:21 ==> Update app: encryption Jul 17 08:59:21 ==> Update app: federatedfilesharing Jul 17 08:59:21 ==> Update app: federation Jul 17 08:59:21 ==> Update app: files Jul 17 08:59:21 ==> Update app: files_external Jul 17 08:59:21 ==> Update app: files_pdfviewer Jul 17 08:59:21 ==> Update app: files_rightclick Jul 17 08:59:21 ==> Update app: files_sharing Jul 17 08:59:21 ==> Update app: files_trashbin Jul 17 08:59:21 ==> Update app: files_versions Jul 17 08:59:21 ==> Update app: firstrunwizard Jul 17 08:59:21 ==> Update app: logreader Jul 17 08:59:21 ==> Update app: lookup_server_connector Jul 17 08:59:21 ==> Update app: nextcloud_announcements Jul 17 08:59:21 ==> Update app: notifications Jul 17 08:59:21 ==> Update app: oauth2 Jul 17 08:59:21 ==> Update app: password_policy Jul 17 08:59:21 ==> Update app: photos Jul 17 08:59:21 ==> Update app: privacy Jul 17 08:59:21 ==> Update app: provisioning_api Jul 17 08:59:21 ==> Update app: recommendations Jul 17 08:59:21 ==> Update app: related_resources Jul 17 08:59:21 ==> Update app: serverinfo Jul 17 08:59:21 ==> Update app: settings Jul 17 08:59:21 ==> Update app: sharebymail Jul 17 08:59:21 ==> Update app: support Jul 17 08:59:21 ==> Update app: survey_client Jul 17 08:59:21 ==> Update app: suspicious_login Jul 17 08:59:21 => Healtheck error: Error: connect ECONNREFUSED 172.18.17.145:80 Jul 17 08:59:21 encryption photos user_ldap Jul 17 08:59:21 federatedfilesharing privacy user_status Jul 17 08:59:21 federation provisioning_api viewer Jul 17 08:59:21 files recommendations weather_status Jul 17 08:59:21 files_external related_resources workflowengine Jul 17 08:59:21 files_pdfviewer serverinfo Jul 17 08:59:21 files_rightclick settings Jul 17 08:59:22 Jul 17 08:59:22 Jul 17 08:59:22 Jul 17 08:59:22 Jul 17 08:59:22 Command "upgrade" is not defined. Jul 17 08:59:22 ==> Copying htaccess Jul 17 08:59:22 ==> New apps: Jul 17 08:59:22 ==> Update app: systemtags Jul 17 08:59:22 ==> Update app: text Jul 17 08:59:22 ==> Update app: theming Jul 17 08:59:22 ==> Update app: twofactor_backupcodes Jul 17 08:59:22 ==> Update app: twofactor_totp Jul 17 08:59:22 ==> Update app: updatenotification Jul 17 08:59:22 ==> Update app: user_ldap Jul 17 08:59:22 ==> Update app: user_status Jul 17 08:59:22 ==> Update app: viewer Jul 17 08:59:22 ==> Update app: weather_status Jul 17 08:59:22 ==> Update app: workflowengine Jul 17 08:59:22 ==> run migration Jul 17 08:59:22 ==> update config Jul 17 08:59:22 Nextcloud is not installed - only a limited number of commands are available Jul 17 08:59:22 accessibility files_sharing sharebymail Jul 17 08:59:22 activity files_sharing settings Jul 17 08:59:22 activity files_trashbin spreed Jul 17 08:59:22 admin_audit files_trashbin sharebymail Jul 17 08:59:22 admin_audit files_versions support Jul 17 08:59:22 bruteforcesettings files_versions support Jul 17 08:59:22 bruteforcesettings files_videoplayer survey_client Jul 17 08:59:22 calendar firstrunwizard suspicious_login Jul 17 08:59:22 circles firstrunwizard survey_client Jul 17 08:59:22 circles logreader systemtags Jul 17 08:59:22 cloud_federation_api logreader suspicious_login Jul 17 08:59:22 cloud_federation_api lookup_server_connector text Jul 17 08:59:22 comments lookup_server_connector systemtags Jul 17 08:59:22 comments nextcloud_announcements theming Jul 17 08:59:22 contactsinteraction nextcloud_announcements text Jul 17 08:59:22 contactsinteraction notifications twofactor_backupcodes Jul 17 08:59:22 dashboard notifications theming Jul 17 08:59:22 dashboard oauth2 twofactor_totp Jul 17 08:59:22 dav oauth2 twofactor_backupcodes Jul 17 08:59:22 dav password_policy updatenotification Jul 17 08:59:22 encryption password_policy twofactor_totp Jul 17 08:59:22 encryption photos user_ldap Jul 17 08:59:22 federatedfilesharing photos updatenotification Jul 17 08:59:22 federatedfilesharing privacy user_status Jul 17 08:59:22 federation privacy user_ldap Jul 17 08:59:22 federation provisioning_api viewer Jul 17 08:59:22 files provisioning_api user_status Jul 17 08:59:22 files recommendations weather_status Jul 17 08:59:22 files_external recommendations viewer Jul 17 08:59:22 files_external related_resources workflowengine Jul 17 08:59:22 files_pdfviewer related_resources weather_status Jul 17 08:59:22 files_pdfviewer serverinfo Jul 17 08:59:22 files_rightclick serverinfo workflowengine Jul 17 08:59:22 files_rightclick settings Jul 17 09:00:14 ==> Updating apps Jul 17 09:00:14 ==> Old apps: Jul 17 09:00:14 activity files_sharing settings Jul 17 09:00:14 admin_audit files_trashbin sharebymail Jul 17 09:00:14 bruteforcesettings files_versions support Jul 17 09:00:14 circles firstrunwizard survey_client Jul 17 09:00:14 cloud_federation_api logreader suspicious_login Jul 17 09:00:14 comments lookup_server_connector systemtags Jul 17 09:00:14 contactsinteraction nextcloud_announcements text Jul 17 09:00:14 dashboard notifications theming Jul 17 09:00:14 dav oauth2 twofactor_backupcodes Jul 17 09:00:14 encryption password_policy twofactor_totp Jul 17 09:00:14 federatedfilesharing photos updatenotification Jul 17 09:00:14 federation privacy user_ldap Jul 17 09:00:14 files provisioning_api user_status Jul 17 09:00:14 files_external recommendations viewer Jul 17 09:00:14 files_pdfviewer related_resources weather_status Jul 17 09:00:14 files_rightclick serverinfo workflowengine Jul 17 09:00:14 accessibility files_sharing sharebymail Jul 17 09:00:14 activity files_trashbin spreed Jul 17 09:00:14 admin_audit files_versions support Jul 17 09:00:14 bruteforcesettings files_videoplayer survey_client Jul 17 09:00:14 calendar firstrunwizard suspicious_login Jul 17 09:00:14 circles logreader systemtags Jul 17 09:00:14 cloud_federation_api lookup_server_connector text Jul 17 09:00:14 comments nextcloud_announcements theming Jul 17 09:00:14 contactsinteraction notifications twofactor_backupcodes Jul 17 09:00:14 dashboard oauth2 twofactor_totp Jul 17 09:00:14 dav password_policy updatenotification Jul 17 09:00:14 encryption photos user_ldap Jul 17 09:00:14 federatedfilesharing privacy user_status Jul 17 09:00:14 federation provisioning_api viewer Jul 17 09:00:14 files recommendations weather_status Jul 17 09:00:14 files_external related_resources workflowengine Jul 17 09:00:14 files_pdfviewer serverinfo Jul 17 09:00:14 files_rightclick settings Jul 17 09:00:14 ==> Update app: activity Jul 17 09:00:14 ==> Update app: admin_audit Jul 17 09:00:14 ==> Update app: bruteforcesettings Jul 17 09:00:14 ==> Update app: circles Jul 17 09:00:14 ==> Update app: cloud_federation_api Jul 17 09:00:14 ==> Update app: comments Jul 17 09:00:14 ==> Update app: contactsinteraction Jul 17 09:00:14 ==> Update app: dashboard Jul 17 09:00:14 ==> Update app: dav Jul 17 09:00:14 ==> Update app: encryption Jul 17 09:00:14 ==> Update app: federatedfilesharing Jul 17 09:00:14 ==> Update app: federation Jul 17 09:00:14 ==> Update app: files Jul 17 09:00:14 ==> Update app: files_external Jul 17 09:00:14 ==> Update app: files_pdfviewer Jul 17 09:00:14 ==> Update app: files_rightclick Jul 17 09:00:14 ==> Update app: files_sharing Jul 17 09:00:14 ==> Update app: files_trashbin Jul 17 09:00:14 ==> Update app: files_versions Jul 17 09:00:14 ==> Update app: firstrunwizard Jul 17 09:00:15 ==> Update app: logreader Jul 17 09:00:15 ==> Update app: lookup_server_connector Jul 17 09:00:15 ==> Update app: nextcloud_announcements Jul 17 09:00:15 ==> Update app: notifications Jul 17 09:00:15 ==> Update app: oauth2 Jul 17 09:00:15 ==> Update app: password_policy Jul 17 09:00:15 ==> Update app: photos Jul 17 09:00:15 ==> Update app: privacy Jul 17 09:00:15 ==> Update app: provisioning_api Jul 17 09:00:15 ==> Update app: recommendations Jul 17 09:00:15 ==> Update app: related_resources Jul 17 09:00:15 ==> Update app: serverinfo Jul 17 09:00:15 ==> Update app: settings Jul 17 09:00:15 ==> Update app: sharebymail Jul 17 09:00:15 ==> Update app: support Jul 17 09:00:15 ==> Update app: survey_client Jul 17 09:00:15 ==> Update app: suspicious_login Jul 17 09:00:15 => Healtheck error: Error: connect ECONNREFUSED 172.18.17.145:80 Jul 17 09:00:15 ==> Update app: systemtags Jul 17 09:00:15 ==> Update app: text Jul 17 09:00:15 ==> Update app: theming Jul 17 09:00:15 ==> Update app: twofactor_backupcodes Jul 17 09:00:15 ==> Update app: twofactor_totp Jul 17 09:00:15 ==> Update app: updatenotification Jul 17 09:00:15 ==> Update app: user_ldap Jul 17 09:00:15 ==> Update app: user_status Jul 17 09:00:15 ==> Update app: viewer Jul 17 09:00:15 ==> Update app: weather_status Jul 17 09:00:15 ==> Update app: workflowengine Jul 17 09:00:15 ==> New apps: Jul 17 09:00:15 activity files_sharing settings Jul 17 09:00:15 admin_audit files_trashbin sharebymail Jul 17 09:00:15 bruteforcesettings files_versions support Jul 17 09:00:15 circles firstrunwizard survey_client Jul 17 09:00:15 cloud_federation_api logreader suspicious_login Jul 17 09:00:15 comments lookup_server_connector systemtags Jul 17 09:00:15 contactsinteraction nextcloud_announcements text Jul 17 09:00:15 dashboard notifications theming Jul 17 09:00:15 dav oauth2 twofactor_backupcodes Jul 17 09:00:15 encryption password_policy twofactor_totp Jul 17 09:00:15 federatedfilesharing photos updatenotification Jul 17 09:00:15 federation privacy user_ldap Jul 17 09:00:15 files provisioning_api user_status Jul 17 09:00:15 files_external recommendations viewer Jul 17 09:00:15 files_pdfviewer related_resources weather_status Jul 17 09:00:15 files_rightclick serverinfo workflowengine Jul 17 09:00:15 accessibility files_sharing sharebymail Jul 17 09:00:15 activity files_trashbin spreed Jul 17 09:00:15 admin_audit files_versions support Jul 17 09:00:15 bruteforcesettings files_videoplayer survey_client Jul 17 09:00:15 calendar firstrunwizard suspicious_login Jul 17 09:00:15 circles logreader systemtags Jul 17 09:00:15 cloud_federation_api lookup_server_connector text Jul 17 09:00:15 comments nextcloud_announcements theming Jul 17 09:00:15 contactsinteraction notifications twofactor_backupcodes Jul 17 09:00:15 dashboard oauth2 twofactor_totp Jul 17 09:00:15 dav password_policy updatenotification Jul 17 09:00:15 encryption photos user_ldap Jul 17 09:00:15 federatedfilesharing privacy user_status Jul 17 09:00:15 federation provisioning_api viewer Jul 17 09:00:15 files recommendations weather_status Jul 17 09:00:15 files_external related_resources workflowengine Jul 17 09:00:15 files_pdfviewer serverinfo Jul 17 09:00:15 files_rightclick settings Jul 17 09:00:15 ==> Copying htaccess Jul 17 09:00:16 ==> update config Jul 17 09:00:16 ==> run migration Jul 17 09:00:16 Nextcloud is not installed - only a limited number of commands are available Jul 17 09:00:16 Jul 17 09:00:16 Jul 17 09:00:16 Command "upgrade" is not defined.
-
@Shai said in After disk full incident, can't restart Nextcloud App:
Jul 17 08:59:22 Nextcloud is not installed - only a limited number of commands are available
This happens mostly when the nextcloud config file gets confused because of out of disk. Can you try this?
- From your previous backup, look for
config/config.php
. This file willinstalled => true
inside it. Replace the config file in your current installation with the one from the backup. - Restart the app
- It should work most likely
If this is tricky, please feel free to reach out to support@cloudron.io
- From your previous backup, look for
-