Nextcloud Internal Server Error

girish

I was hoping to find the list of default apps in nextcloud docs but couldn't find it. The default is below. Cloudron only install the ldap app extra.

root@329ac7c9-1419-4f18-bff2-3cbe0743d8d0:/app/code# sudo -u www-data php -f /app/code/occ app:list
Enabled:
  - accessibility: 1.4.0
  - activity: 2.11.0
  - cloud_federation_api: 1.1.0
  - comments: 1.8.0
  - dav: 1.14.0
  - federatedfilesharing: 1.8.0
  - federation: 1.8.0
  - files: 1.13.1
  - files_pdfviewer: 1.7.0
  - files_rightclick: 0.15.2
  - files_sharing: 1.10.1
  - files_trashbin: 1.8.0
  - files_versions: 1.11.0
  - files_videoplayer: 1.7.0
  - firstrunwizard: 2.7.0
  - logreader: 2.3.0
  - lookup_server_connector: 1.6.0
  - nextcloud_announcements: 1.7.0
  - notifications: 2.6.0
  - oauth2: 1.6.0
  - password_policy: 1.8.0
  - photos: 1.0.0
  - privacy: 1.2.0
  - provisioning_api: 1.8.0
  - recommendations: 0.6.0
  - serverinfo: 1.8.0
  - settings: 1.0.0
  - sharebymail: 1.8.0
  - spreed: 8.0.8
  - support: 1.1.0
  - survey_client: 1.6.0
  - systemtags: 1.8.0
  - text: 2.0.0
  - theming: 1.9.0
  - twofactor_backupcodes: 1.7.0
  - updatenotification: 1.8.0
  - user_ldap: 1.8.0
  - viewer: 1.2.0
  - workflowengine: 2.0.0
Disabled:
  - admin_audit
  - encryption
  - files_external

girish

To add, the answer is that all the apps under apps/ are pre-installed as per this comment in code.

So, https://github.com/nextcloud/server/tree/master/apps is the list

avatar1024

Hello,

I have 3 cloudron instances with nextcloud. One of them has started to show the Internal Server Error message when I try to login (first time about two weeks ago).

Restarting the app container or whole server doesn't work. The way I could bring it back was from a backup. But it's done it again about a week ago (so I restored again from a backup that was done in the morning that day), and then the error occurred again today.

Files_external is not installed.
All data are in Group Folders so disable the group folders app does bring back nextcloud but then I have no data. One could think it is the Group Folder app the problem but then why would it work from back with the exact same version? I feel it is actually to do with some errors in the data but I'm not knowledgeable enough to understand.

Here is part of the log:

Jul 30 17:16:23 [Fri Jul 30 15:16:23.839172 2021] [php7:notice] [pid 3283] [client 172.18.0.1:60162] {"reqId":"PHGXkYj36DZVzPDYZLUq","level":3,"time":"2021-07-30T15:16:23+00:00","remoteAddr":"188.241.83.109","user":"ggn","app":"core","method":"GET","url":"/js/files/merged-index.js?v=d548c334-0","message":"Following symlinks is not allowed ('/app/data//core' -> '/app/code/core/' not inside '/app/data/')","userAgent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:90.0) Gecko/20100101 Firefox/90.0","version":"21.0.3.1"}, referer: https://files.xxxxx.org.uk/apps/files/
Jul 30 17:16:23 [Fri Jul 30 15:16:23.840876 2021] [php7:notice] [pid 3283] [client 172.18.0.1:60162] {"reqId":"PHGXkYj36DZVzPDYZLUq","level":3,"time":"2021-07-30T15:16:23+00:00","remoteAddr":"188.241.83.109","user":"ggn","app":"index","method":"GET","url":"/js/files/merged-index.js?v=d548c334-0","message":{"Exception":"OCP\\\\Files\\\\ForbiddenException","Message":"Following symlinks is not allowed","Code":0,"Trace":[{"file":"/app/code/lib/private/Files/Storage/Local.php","line":150,"function":"getSourcePath","class":"OC\\\\Files\\\\Storage\\\\Local","type":"->"},{"file":"/app/code/lib/private/Files/Storage/Local.php","line":165,"function":"stat","class":"OC\\\\Files\\\\Storage\\\\Local","type":"->"},{"file":"/app/code/lib/private/Files/Storage/Common.php","line":884,"function":"getMetaData","class":"OC\\\\Files\\\\Storage\\\\Local","type":"->"},{"function":"getDirectoryContent","class":"OC\\\\Files\\\\Storage\\\\Common","type":"->"},{"file":"/app/code/lib/private/Files/Cache/Scanner.php","line":409,"function":"iterator_to_array"},{"file":"/app/code/lib/private/Files/Cache/Scanner.php","line":389,"function":"handleChildren","class":"OC\\\\Files\\\\Cache\\\\Scanner","type":"->"},{"file":"/app/code/lib/private/Files/Cache/Scanner.php","line":341,"function":"scanChildren","class":"OC\\\\Files\\\\Cache\\\\Scanner","type":"->"},{"file":"/app/code/lib/private/Files/Cache/LocalRootScanner.php","line":40,"function":"scan","class":"OC\\\\Files\\\\Cache\\\\Scanner","type":"->"},{"file":"/app/code/lib/private/Files/View.php","line":1350,"function":"scan","class":"OC\\\\Files\\\\Cache\\\\LocalRootScanner","type":"->"},{"file":"/app/code/lib/private/Files/View.php","line":1394,"function":"getCacheEntry","class":"OC\\\\Files\\\\View","type":"->"},{"file":"/app/code/lib/private/Files/Node/Node.php","line":100,"function":"getFileInfo","class":"OC\\\\Files\\\\View","type":"->"},{"file":"/app/code/lib/private/Files/Node/Node.php","line":362,"function":"getFileInfo","class":"OC\\\\Files\\\\Node\\\\Node","type":"->"},{"function":"getMountPoint","class":"OC\\\\Files\\\\Node\\\\Node","type":"->"},{"file":"/app/code/lib/private/Files/Node/LazyFolder.php","line":66,"function":"call_user_func_array"},{"file":"/app/code/lib/private/Files/Node/LazyFolder.php","line":332,"function":"__call","class":"OC\\\\Files\\\\Node\\\\LazyFolder","type":"->"},{"file":"/app/data/apps/groupfolders/lib/ACL/ACLManager.php","line":49,"function":"getMountPoint","class":"OC\\\\Files\\\\Node\\\\LazyFolder","type":"->"},{"file":"/app/data/apps/groupfolders/lib/ACL/ACLManager.php","line":75,"function":"getRootStorageId","class":"OCA\\\\GroupFolders\\\\ACL\\\\ACLManager","type":"->"},{"file":"/app/data/apps/groupfolders/lib/ACL/ACLManager.php","line":113,"function":"getRules","class":"OCA\\\\GroupFolders\\\\ACL\\\\ACLManager","type":"->"},{"file":"/app/data/apps/groupfolders/lib/Mount/MountProvider.php","line":176,"function":"getACLPermissionsForPath","class":"OCA\\\\GroupFolders\\\\ACL\\\\ACLManager","type":"->"},{"file":"/app/data/apps/groupfolders/lib/Mount/MountProvider.php","line":122,"function":"getMount","class":"OCA\\\\GroupFolders\\\\Mount\\\\MountProvider","type":"->"},{"function":"OCA\\\\GroupFolders\\\\Mount\\\\{closure}","class":"OCA\\\\GroupFolders\\\\Mount\\\\MountProvider","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/app/data/apps/groupfolders/lib/Mount/MountProvider.php","line":102,"function":"array_map"},{"file":"/app/code/lib/private/Files/Config/MountProviderCollection.php","line":109,"function":"getMountsForUser","class":"OCA\\\\GroupFolders\\\\Mount\\\\MountProvider","type":"->"},{"file":"/app/code/lib/private/Files/Filesystem.php","line":452,"function":"addMountForUser","class":"OC\\\\Files\\\\Config\\\\MountProviderCollection","type":"->"},{"file":"/app/code/lib/private/Files/Filesystem.php","line":377,"function":"initMountPoints","class":"OC\\\\Files\\\\Filesystem","type":"::"},{"file":"/app/code/lib/private/legacy/OC_Util.php","line":324,"function":"init","class":"OC\\\\Files\\\\Filesystem","type":"::"},{"file":"/app/code/lib/private/Files/Mount/Manager.php","line":92,"function":"setupFS","class":"OC_Util","type":"::"},{"file":"/app/code/lib/private/Files/View.php","line":1386,"function":"find","class":"OC\\\\Files\\\\Mount\\\\Manager","type":"->"},{"file":"/app/code/lib/private/Files/Node/Root.php","line":201,"function":"getFileInfo","class":"OC\\\\Files\\\\View","type":"->"},{"function":"get","class":"OC\\\\Files\\\\Node\\\\Root","type":"->"},{"file":"/app/code/lib/private/Files/Node/LazyFolder.php","line":66,"function":"call_user_func_array"},{"file":"/app/code/lib/private/Files/Node/LazyFolder.php","line":143,"function":"__call","class":"OC\\\\Files\\\\Node\\\\LazyFolder","type":"->"},{"file":"/app/code/lib/private/Files/AppData/AppData.php","line":142,"function":"get","class":"OC\\\\Files\\\\Node\\\\LazyFolder","type":"->"},{"file":"/app/code/core/Controller/JsController.php","line":73,"function":"getFolder","class":"OC\\\\Files\\\\AppData\\\\AppData","type":"->"},{"file":"/app/code/lib/private/AppFramework/Http/Dispatcher.php","line":218,"function":"getJs","class":"OC\\\\Core\\\\Controller\\\\JsController","type":"->"},{"file":"/app/code/lib/private/AppFramework/Http/Dispatcher.php","line":127,"function":"executeController","class":"OC\\\\AppFramework\\\\Http\\\\Dispatcher","type":"->"},{"file":"/app/code/lib/private/AppFramework/App.php","line":157,"function":"dispatch","class":"OC\\\\AppFramework\\\\Http\\\\Dispatcher","type":"->"},{"file":"/app/code/lib/private/Route/Router.php","line":302,"function":"main","class":"OC\\\\AppFramework\\\\App","type":"::"},{"file":"/app/code/lib/base.php","line":993,"function":"match","class":"OC\\\\Route\\\\Router","type":"->"},{"file":"/app/code/index.php","line":37,"function":"handleRequest","class":"OC","type":"::"}],"File":"/app/code/lib/private/Files/Storage/Local.php","Line":474,"CustomMessage":"--"},"userAgent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:90.0) Gecko/20100101

Could anyone help??

girish

@avatar1024 It seems the error is coming form the groupfolders app - https://apps.nextcloud.com/apps/groupfolders . Can you try disabling that app? See https://docs.cloudron.io/apps/nextcloud/#fixing-a-broken-install for instructions on how to disable it.

girish

@avatar1024 Whoops, I didn't read your entire report. The error message seems to be Following symlinks is not allowed. Let me see if it's anything obvious.

girish

@avatar1024 If you can revert the app to a working state, then can you tell me the version of the nextcloud package (in cloudron dashboard, updates view of nextcloud) and also the version of the groupfolder app? Maybe one of these is auto-updating and causing some breakage.

Also, while we try to resolve this, maybe you can disable automatic updates for nextcloud.

avatar1024

@girish Hi Girish, thank you for your reply. Sorry I had an emergency and could not reply earlier.

Nextcloud version is 21.0.3 (package 4.9.2) and was updated 5 weeks ago. So this issue started some time after that update. However when I restore to a working state the version is the same.

Basically what happens is that it all works fine, then suddenly I get this error, then I restore from backup that is done the same morning and it works fine again. Obviously this is not ideal as there is potential data loss during the day.

Group folder app version is 9.0.2, again no version change between the working / not working state.

Due to a mistake of mine with backup retention policy and having had to leave my computer for most of the weekend, I now cannot recover a recent working version of the app. Do you know if I restore from an old backup if nextcloud will sync automatically any local files that are more recent into the server or will the local files get deleted?

Also when I know tried to restore from a recent backup that was in maintenance mode, I get a "not responding" state and the terminal just loops and doesn't start so I cannot enter a command to get out of maintenance mode. Any way around that?

avatar1024

@avatar1024 As an update: I have restored to working state from a two weeks old backup (but I backed up the data manually from the data folder on the server so hopefully no data will actually be lost). I do not get any more simlinks errors in the log so I wonder if that is one of the issue?

avatar1024

@avatar1024 So I've copied the "/data" folder from the recent but broken app version into the app directory of two weeks old but working version, did a

sudo -u www-data php -f /app/code/occ files:scan --all

and it seems to all be working but all recent data.

I obviously just did a new backup but still it would be good to understand what caused the Internal Server Error as I expect it'll break again as it has been happening weekly for the past two weeks.

girish

@avatar1024 For a start, can you disable automatic updates of nextcloud in Cloudron ? That will get us a good idea as to what actually causes the problem. Also, make sure that you do not update the apps inside nextcloud for some time as well. Let's see if the problem can be reproduced when the versions are static.

avatar1024

@girish Sure, I've disabled the update

girish

@avatar1024 OK, let's wait and see if it becomes reproducible

avatar1024

@girish Ok I'll let you know if it happens again. Anything I should be watching out for when it happens again to help figuring out where the issue comes from?

Regarding version: the first time the issue happened was on 14 July so a good week after the nextcoud upgrade to 21.0.3 (package 4.9.2) and almost a month after the upgrade of the group folder app to 9.0.2. In between three times the issue happened there was no version change (or upgrade) of either the nextcloud package or the group folder app.

micmc

@girish said in Nextcloud Internal Server Error:

@avatar1024 It seems the error is coming form the groupfolders app - https://apps.nextcloud.com/apps/groupfolders . Can you try disabling that app? See https://docs.cloudron.io/apps/nextcloud/#fixing-a-broken-install for instructions on how to disable it.

Hi Girish I'm getting the same type of error as the app refuses to respond and event restart with apparently not much left to do than restore a previous working version as @avatar1024 have done.

This is occurring after the latest update and I'm now running Nextcloud 22.0.0 com.nextcloud.cloudronapp@4.10.0 .

It's acting in the like of avatar1024 however my 'php notice' line is much less long and seems to be referring to something which relates to Group Manager as you pointed out.

Here's one line :

Aug 06 08:11:50 [Fri Aug 06 12:11:50.409710 2021] [php7:notice] [pid 184] [client 172.18.0.1:35504] {"reqId":"6sa9ysMrCxhK","level":3,"time":"2021-08-06T12:11:50+00:00","remoteAddr":"172.18.0.1","user":"--","app":"index","method":"GET","url":"/index.php/login","message":"Argument 1 passed to OC\\\\Group\\\\Manager::getUserIdGroupIds() must be of the type string, null given, called in /app/code/lib/private/Group/Manager.php on line 316","userAgent":"Mozilla (CloudronHealth)","version":"22.0.0.11","exception":{"Exception":"TypeError","Message":"Argument 1 passed to OC\\\\Group\\\\Manager::getUserIdGroupIds() must be of the type string, null given, called in /app/code/lib/private/Group/Manager.php on line 316","Code":0,"Trace":[{"file":"/app/code/lib/private/Group/Manager.php","line":316,"function":"getUserIdGroupIds","class":"OC\\\\Group\\\\Manager","type":"->"},{"file":"/app/data/apps/deck/lib/Service/PermissionService.php","line":316,"function":"isInGroup","class":"OC\\\\Group\\\\Manager","type":"->"},{"file":"/app/data/apps/deck/lib/Capabilities.php","line":53,"function":"canCreate","class":"OCA\\\\Deck\\\\Service\\\\PermissionService","type":"->"},{"file":"/app/code/lib/private/CapabilitiesManager.php","line":69,"function":"getCapabilities","class":"OCA\\\\Deck\\\\Capabilities","type":"->"},{"file":"/app/code/lib/private/Template/JSConfigHelper.php","line":190,"function":"getCapabilities","class":"OC\\\\CapabilitiesManager","type":"->"},{"file":"/app/code/lib/private/TemplateLayout.php","line":220,"function":"getConfig","class":"OC\\\\Template\\\\JSConfigHelper","type":"->"},{"file":"/app/code/lib/private/legacy/OC_Template.php","line":182,"function":"__construct","class":"OC\\\\TemplateLayout","type":"->"},{"file":"/app/code/lib/public/AppFramework/Http/TemplateResponse.php","line":204,"function":"fetchPage","class":"OC_Template","type":"->"},{"file":"/app/code/lib/private/AppFramework/Http/Dispatcher.php","line":171,"function":"render","class":"OCP\\\\AppFramework\\\\Http\\\\TemplateResponse","type":"->"},{"file":"/app/code/lib/private/AppFramework/App.php","line":156,"function":"dispatch","class":"OC\\\\AppFramework\\\\Http\\\\Dispatcher","type":"->"},{"file":"/app/code/lib/private/Route/Router.php","line":301,"function":"main","class":"OC\\\\AppFramework\\\\App","type":"::"},{"file":"/app/code/lib/base.php","line":1000,"function":"match","class":"OC\\\\Route\\\\Router","type":"->"},{"file":"/app/code/index.php","line":36,"function":"handleRequest","class":"OC","type":"::"}],"File":"/app/code/lib/private/Group/Manager.php","Line":333,"CustomMessage":"--"}}

I've tried all is suggested in troubleshooting, recovery mode, terminal and start.sh in the docs and what you suggested here as well, and now I've disabled automated updates as well.

Finally, I tried to disable the groupfolders via terminal however, as I expected, it is already disabled. Then, to see the outcome, I tried to enable it and got the message that it could not as it is "not compatible with this version of the server".

Trying to restart yet another time ; "Not responding" end of transmission loll

Andy

micmc

@girish Amendment

After reviewing the php:notice line more carefully I noticed a reference to Deck, which is one app I'd enabled.

So I went back to terminal to disable Deck and boom, the app restart automtically and now runs as well oiled. Wow. That was it bro.

Andy

rmdes

got the same issue with one of my nextcloud's, even after removing the app, I could not restore the app to normal state, in the end, I noticed Circles app was throwing errors, disabled with

sudo -u www-data php occ app:list

sudo -u www-data php occ app:disable circles

Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.

Cloudron Forum

Nextcloud Internal Server Error