Nextcloud Internal Server Error

[jdaviescoates]

@christiaan have you tried completely deleting the relevant folder?

[christiaan]

@jdaviescoates this one?

/app/data/apps/files_external_dropbox/

Just deleted now and restarted container without success.

[jdaviescoates]

@christiaan perhaps time to go back to your last backup and restore that?

[christiaan]

I started disabling every third party app and then reloading URL each time and this one solved my problem:

sudo -u www-data php occ app:disable files_external

I don't remember actively installing that one. Isn't that on by default? Or maybe it was a default app (you can't remove it) but it wasn't enabled and I enabled it.

Thanks for the help @jdaviescoates

[christiaan]

This post is deleted!

[christiaan]

This post is deleted!

[girish]

@christiaan I think you bring up a good point about what apps are pre-configured/pre-installed in the package is not clear up front. I will add this to our docs because it helps in debugging like in your case.

On a side note, Nextcloud apps are a constant source of frustration during updates. So, it's best to install as few as possible. I even put this note in our docs (https://cloudron.io/documentation/apps/nextcloud/#email) :

We do not recommend installing apps in Nextcloud unless absolutely 
required. Maintaining such systems is a security hassle since you
need to keep them up-to-date. Apps often break when Nextcloud is
updated and you have to know how to fix them. Finally, Apps are not
run  sandboxed. This means that a faulty plugin might compromise the
whole app and also not make the app work at all.

[girish]

I was hoping to find the list of default apps in nextcloud docs but couldn't find it. The default is below. Cloudron only install the ldap app extra.

root@329ac7c9-1419-4f18-bff2-3cbe0743d8d0:/app/code# sudo -u www-data php -f /app/code/occ app:list
Enabled:
  - accessibility: 1.4.0
  - activity: 2.11.0
  - cloud_federation_api: 1.1.0
  - comments: 1.8.0
  - dav: 1.14.0
  - federatedfilesharing: 1.8.0
  - federation: 1.8.0
  - files: 1.13.1
  - files_pdfviewer: 1.7.0
  - files_rightclick: 0.15.2
  - files_sharing: 1.10.1
  - files_trashbin: 1.8.0
  - files_versions: 1.11.0
  - files_videoplayer: 1.7.0
  - firstrunwizard: 2.7.0
  - logreader: 2.3.0
  - lookup_server_connector: 1.6.0
  - nextcloud_announcements: 1.7.0
  - notifications: 2.6.0
  - oauth2: 1.6.0
  - password_policy: 1.8.0
  - photos: 1.0.0
  - privacy: 1.2.0
  - provisioning_api: 1.8.0
  - recommendations: 0.6.0
  - serverinfo: 1.8.0
  - settings: 1.0.0
  - sharebymail: 1.8.0
  - spreed: 8.0.8
  - support: 1.1.0
  - survey_client: 1.6.0
  - systemtags: 1.8.0
  - text: 2.0.0
  - theming: 1.9.0
  - twofactor_backupcodes: 1.7.0
  - updatenotification: 1.8.0
  - user_ldap: 1.8.0
  - viewer: 1.2.0
  - workflowengine: 2.0.0
Disabled:
  - admin_audit
  - encryption
  - files_external

[girish]

To add, the answer is that all the apps under apps/ are pre-installed as per this comment in code.

So, https://github.com/nextcloud/server/tree/master/apps is the list

[avatar1024]

Hello,

I have 3 cloudron instances with nextcloud. One of them has started to show the Internal Server Error message when I try to login (first time about two weeks ago).

Restarting the app container or whole server doesn't work. The way I could bring it back was from a backup. But it's done it again about a week ago (so I restored again from a backup that was done in the morning that day), and then the error occurred again today.

Files_external is not installed.
All data are in Group Folders so disable the group folders app does bring back nextcloud but then I have no data. One could think it is the Group Folder app the problem but then why would it work from back with the exact same version? I feel it is actually to do with some errors in the data but I'm not knowledgeable enough to understand.

Here is part of the log:

Jul 30 17:16:23 [Fri Jul 30 15:16:23.839172 2021] [php7:notice] [pid 3283] [client 172.18.0.1:60162] {"reqId":"PHGXkYj36DZVzPDYZLUq","level":3,"time":"2021-07-30T15:16:23+00:00","remoteAddr":"188.241.83.109","user":"ggn","app":"core","method":"GET","url":"/js/files/merged-index.js?v=d548c334-0","message":"Following symlinks is not allowed ('/app/data//core' -> '/app/code/core/' not inside '/app/data/')","userAgent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:90.0) Gecko/20100101 Firefox/90.0","version":"21.0.3.1"}, referer: https://files.xxxxx.org.uk/apps/files/
Jul 30 17:16:23 [Fri Jul 30 15:16:23.840876 2021] [php7:notice] [pid 3283] [client 172.18.0.1:60162] {"reqId":"PHGXkYj36DZVzPDYZLUq","level":3,"time":"2021-07-30T15:16:23+00:00","remoteAddr":"188.241.83.109","user":"ggn","app":"index","method":"GET","url":"/js/files/merged-index.js?v=d548c334-0","message":{"Exception":"OCP\\\\Files\\\\ForbiddenException","Message":"Following symlinks is not allowed","Code":0,"Trace":[{"file":"/app/code/lib/private/Files/Storage/Local.php","line":150,"function":"getSourcePath","class":"OC\\\\Files\\\\Storage\\\\Local","type":"->"},{"file":"/app/code/lib/private/Files/Storage/Local.php","line":165,"function":"stat","class":"OC\\\\Files\\\\Storage\\\\Local","type":"->"},{"file":"/app/code/lib/private/Files/Storage/Common.php","line":884,"function":"getMetaData","class":"OC\\\\Files\\\\Storage\\\\Local","type":"->"},{"function":"getDirectoryContent","class":"OC\\\\Files\\\\Storage\\\\Common","type":"->"},{"file":"/app/code/lib/private/Files/Cache/Scanner.php","line":409,"function":"iterator_to_array"},{"file":"/app/code/lib/private/Files/Cache/Scanner.php","line":389,"function":"handleChildren","class":"OC\\\\Files\\\\Cache\\\\Scanner","type":"->"},{"file":"/app/code/lib/private/Files/Cache/Scanner.php","line":341,"function":"scanChildren","class":"OC\\\\Files\\\\Cache\\\\Scanner","type":"->"},{"file":"/app/code/lib/private/Files/Cache/LocalRootScanner.php","line":40,"function":"scan","class":"OC\\\\Files\\\\Cache\\\\Scanner","type":"->"},{"file":"/app/code/lib/private/Files/View.php","line":1350,"function":"scan","class":"OC\\\\Files\\\\Cache\\\\LocalRootScanner","type":"->"},{"file":"/app/code/lib/private/Files/View.php","line":1394,"function":"getCacheEntry","class":"OC\\\\Files\\\\View","type":"->"},{"file":"/app/code/lib/private/Files/Node/Node.php","line":100,"function":"getFileInfo","class":"OC\\\\Files\\\\View","type":"->"},{"file":"/app/code/lib/private/Files/Node/Node.php","line":362,"function":"getFileInfo","class":"OC\\\\Files\\\\Node\\\\Node","type":"->"},{"function":"getMountPoint","class":"OC\\\\Files\\\\Node\\\\Node","type":"->"},{"file":"/app/code/lib/private/Files/Node/LazyFolder.php","line":66,"function":"call_user_func_array"},{"file":"/app/code/lib/private/Files/Node/LazyFolder.php","line":332,"function":"__call","class":"OC\\\\Files\\\\Node\\\\LazyFolder","type":"->"},{"file":"/app/data/apps/groupfolders/lib/ACL/ACLManager.php","line":49,"function":"getMountPoint","class":"OC\\\\Files\\\\Node\\\\LazyFolder","type":"->"},{"file":"/app/data/apps/groupfolders/lib/ACL/ACLManager.php","line":75,"function":"getRootStorageId","class":"OCA\\\\GroupFolders\\\\ACL\\\\ACLManager","type":"->"},{"file":"/app/data/apps/groupfolders/lib/ACL/ACLManager.php","line":113,"function":"getRules","class":"OCA\\\\GroupFolders\\\\ACL\\\\ACLManager","type":"->"},{"file":"/app/data/apps/groupfolders/lib/Mount/MountProvider.php","line":176,"function":"getACLPermissionsForPath","class":"OCA\\\\GroupFolders\\\\ACL\\\\ACLManager","type":"->"},{"file":"/app/data/apps/groupfolders/lib/Mount/MountProvider.php","line":122,"function":"getMount","class":"OCA\\\\GroupFolders\\\\Mount\\\\MountProvider","type":"->"},{"function":"OCA\\\\GroupFolders\\\\Mount\\\\{closure}","class":"OCA\\\\GroupFolders\\\\Mount\\\\MountProvider","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/app/data/apps/groupfolders/lib/Mount/MountProvider.php","line":102,"function":"array_map"},{"file":"/app/code/lib/private/Files/Config/MountProviderCollection.php","line":109,"function":"getMountsForUser","class":"OCA\\\\GroupFolders\\\\Mount\\\\MountProvider","type":"->"},{"file":"/app/code/lib/private/Files/Filesystem.php","line":452,"function":"addMountForUser","class":"OC\\\\Files\\\\Config\\\\MountProviderCollection","type":"->"},{"file":"/app/code/lib/private/Files/Filesystem.php","line":377,"function":"initMountPoints","class":"OC\\\\Files\\\\Filesystem","type":"::"},{"file":"/app/code/lib/private/legacy/OC_Util.php","line":324,"function":"init","class":"OC\\\\Files\\\\Filesystem","type":"::"},{"file":"/app/code/lib/private/Files/Mount/Manager.php","line":92,"function":"setupFS","class":"OC_Util","type":"::"},{"file":"/app/code/lib/private/Files/View.php","line":1386,"function":"find","class":"OC\\\\Files\\\\Mount\\\\Manager","type":"->"},{"file":"/app/code/lib/private/Files/Node/Root.php","line":201,"function":"getFileInfo","class":"OC\\\\Files\\\\View","type":"->"},{"function":"get","class":"OC\\\\Files\\\\Node\\\\Root","type":"->"},{"file":"/app/code/lib/private/Files/Node/LazyFolder.php","line":66,"function":"call_user_func_array"},{"file":"/app/code/lib/private/Files/Node/LazyFolder.php","line":143,"function":"__call","class":"OC\\\\Files\\\\Node\\\\LazyFolder","type":"->"},{"file":"/app/code/lib/private/Files/AppData/AppData.php","line":142,"function":"get","class":"OC\\\\Files\\\\Node\\\\LazyFolder","type":"->"},{"file":"/app/code/core/Controller/JsController.php","line":73,"function":"getFolder","class":"OC\\\\Files\\\\AppData\\\\AppData","type":"->"},{"file":"/app/code/lib/private/AppFramework/Http/Dispatcher.php","line":218,"function":"getJs","class":"OC\\\\Core\\\\Controller\\\\JsController","type":"->"},{"file":"/app/code/lib/private/AppFramework/Http/Dispatcher.php","line":127,"function":"executeController","class":"OC\\\\AppFramework\\\\Http\\\\Dispatcher","type":"->"},{"file":"/app/code/lib/private/AppFramework/App.php","line":157,"function":"dispatch","class":"OC\\\\AppFramework\\\\Http\\\\Dispatcher","type":"->"},{"file":"/app/code/lib/private/Route/Router.php","line":302,"function":"main","class":"OC\\\\AppFramework\\\\App","type":"::"},{"file":"/app/code/lib/base.php","line":993,"function":"match","class":"OC\\\\Route\\\\Router","type":"->"},{"file":"/app/code/index.php","line":37,"function":"handleRequest","class":"OC","type":"::"}],"File":"/app/code/lib/private/Files/Storage/Local.php","Line":474,"CustomMessage":"--"},"userAgent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:90.0) Gecko/20100101 

Could anyone help??

[girish]

@avatar1024 It seems the error is coming form the groupfolders app - https://apps.nextcloud.com/apps/groupfolders . Can you try disabling that app? See https://docs.cloudron.io/apps/nextcloud/#fixing-a-broken-install for instructions on how to disable it.

[girish]

@avatar1024 Whoops, I didn't read your entire report. The error message seems to be Following symlinks is not allowed. Let me see if it's anything obvious.

[girish]

@avatar1024 If you can revert the app to a working state, then can you tell me the version of the nextcloud package (in cloudron dashboard, updates view of nextcloud) and also the version of the groupfolder app? Maybe one of these is auto-updating and causing some breakage.

Also, while we try to resolve this, maybe you can disable automatic updates for nextcloud.

[avatar1024]

@girish Hi Girish, thank you for your reply. Sorry I had an emergency and could not reply earlier.

Nextcloud version is 21.0.3 (package 4.9.2) and was updated 5 weeks ago. So this issue started some time after that update. However when I restore to a working state the version is the same.

Basically what happens is that it all works fine, then suddenly I get this error, then I restore from backup that is done the same morning and it works fine again. Obviously this is not ideal as there is potential data loss during the day.

Group folder app version is 9.0.2, again no version change between the working / not working state.

Due to a mistake of mine with backup retention policy and having had to leave my computer for most of the weekend, I now cannot recover a recent working version of the app. Do you know if I restore from an old backup if nextcloud will sync automatically any local files that are more recent into the server or will the local files get deleted?

Also when I know tried to restore from a recent backup that was in maintenance mode, I get a "not responding" state and the terminal just loops and doesn't start so I cannot enter a command to get out of maintenance mode. Any way around that?

[avatar1024]

@avatar1024 As an update: I have restored to working state from a two weeks old backup (but I backed up the data manually from the data folder on the server so hopefully no data will actually be lost). I do not get any more simlinks errors in the log so I wonder if that is one of the issue?

[avatar1024]

@avatar1024 So I've copied the "/data" folder from the recent but broken app version into the app directory of two weeks old but working version, did a

sudo -u www-data php -f /app/code/occ files:scan --all

and it seems to all be working but all recent data.

I obviously just did a new backup but still it would be good to understand what caused the Internal Server Error as I expect it'll break again as it has been happening weekly for the past two weeks.

[girish]

@avatar1024 For a start, can you disable automatic updates of nextcloud in Cloudron ? That will get us a good idea as to what actually causes the problem. Also, make sure that you do not update the apps inside nextcloud for some time as well. Let's see if the problem can be reproduced when the versions are static.

[avatar1024]

@girish Sure, I've disabled the update

[girish]

@avatar1024 OK, let's wait and see if it becomes reproducible

[avatar1024]

@girish Ok I'll let you know if it happens again. Anything I should be watching out for when it happens again to help figuring out where the issue comes from?

Regarding version: the first time the issue happened was on 14 July so a good week after the nextcoud upgrade to 21.0.3 (package 4.9.2) and almost a month after the upgrade of the group folder app to 9.0.2. In between three times the issue happened there was no version change (or upgrade) of either the nextcloud package or the group folder app.