filescan errors

SolarSimon

Hi there,

When I run "occ files:scan --all" I get quite some errors. They exist since some time, but I had no problems in daylie usage. And searching didnt really get me somewhere.
Does someone have a clue where to start and how to solve them?

First it says, for some users, their Folders don't exist. But these users don't have problems.

Path not found: /username

And then there is a bunch of errors, where I really don't know where to start.

Exception during scan: An exception occurred while executing a query: SQLSTATE[25P02]: In failed sql transaction: 7 ERROR:  current transaction is aborted, commands ignored until end of transaction block
#0 /app/code/3rdparty/doctrine/dbal/src/Connection.php(1938): Doctrine\DBAL\Driver\API\PostgreSQL\ExceptionConverter->convert()
#1 /app/code/3rdparty/doctrine/dbal/src/Connection.php(1880): Doctrine\DBAL\Connection->handleDriverException()
#2 /app/code/3rdparty/doctrine/dbal/src/Connection.php(1105): Doctrine\DBAL\Connection->convertExceptionDuringQuery()
#3 /app/code/lib/private/DB/Connection.php(261): Doctrine\DBAL\Connection->executeQuery()
#4 /app/code/3rdparty/doctrine/dbal/src/Query/QueryBuilder.php(341): OC\DB\Connection->executeQuery()
#5 /app/code/3rdparty/doctrine/dbal/src/Query/QueryBuilder.php(377): Doctrine\DBAL\Query\QueryBuilder->executeQuery()
#6 /app/code/lib/private/DB/QueryBuilder/QueryBuilder.php(280): Doctrine\DBAL\Query\QueryBuilder->execute()
#7 /app/code/lib/private/Files/Cache/Cache.php(172): OC\DB\QueryBuilder\QueryBuilder->execute()
#8 /app/code/lib/private/Files/Cache/HomeCache.php(57): OC\Files\Cache\Cache->get()
#9 /app/code/lib/private/Files/Cache/Cache.php(553): OC\Files\Cache\HomeCache->get()
#10 /app/code/lib/private/Files/Cache/Scanner.php(286): OC\Files\Cache\Cache->remove()
#11 /app/code/lib/private/Files/Cache/Scanner.php(520): OC\Files\Cache\Scanner->removeFromCache()
#12 /app/code/lib/private/Files/Cache/Scanner.php(404): OC\Files\Cache\Scanner->handleChildren()
#13 /app/code/lib/private/Files/Cache/Scanner.php(409): OC\Files\Cache\Scanner->scanChildren()
#14 /app/code/lib/private/Files/Cache/Scanner.php(409): OC\Files\Cache\Scanner->scanChildren()
#15 /app/code/lib/private/Files/Cache/Scanner.php(409): OC\Files\Cache\Scanner->scanChildren()
#16 /app/code/lib/private/Files/Cache/Scanner.php(409): OC\Files\Cache\Scanner->scanChildren()
#17 /app/code/lib/private/Files/Cache/Scanner.php(354): OC\Files\Cache\Scanner->scanChildren()
#18 /app/code/lib/private/Files/Utils/Scanner.php(277): OC\Files\Cache\Scanner->scan()
#19 /app/data/apps/files/lib/Command/Scan.php(180): OC\Files\Utils\Scanner->scan()
#20 /app/data/apps/files/lib/Command/Scan.php(241): OCA\Files\Command\Scan->scanFiles()
#21 /app/code/3rdparty/symfony/console/Command/Command.php(298): OCA\Files\Command\Scan->execute()
#22 /app/code/core/Command/Base.php(177): Symfony\Component\Console\Command\Command->run()
#23 /app/code/3rdparty/symfony/console/Application.php(1040): OC\Core\Command\Base->run()
#24 /app/code/3rdparty/symfony/console/Application.php(301): Symfony\Component\Console\Application->doRunCommand()
#25 /app/code/3rdparty/symfony/console/Application.php(171): Symfony\Component\Console\Application->doRun()
#26 /app/code/lib/private/Console/Application.php(213): Symfony\Component\Console\Application->run()
#27 /app/code/console.php(100): OC\Console\Application->run()
#28 /app/code/occ(11): require_once('...')
#29 {main}

In ExceptionConverter.php line 87:
                                                                                                                                                                                        
  An exception occurred while executing a query: SQLSTATE[25P02]: In failed sql transaction: 7 ERROR:  current transaction is aborted, commands ignored until end of transaction block  
                                                                                                                                                                                        

In Exception.php line 28:
                                                                                                                                         
  SQLSTATE[25P02]: In failed sql transaction: 7 ERROR:  current transaction is aborted, commands ignored until end of transaction block  
                                                                                                                                         

In Statement.php line 130:
                                                                                                                                         
  SQLSTATE[25P02]: In failed sql transaction: 7 ERROR:  current transaction is aborted, commands ignored until end of transaction block

I am running v7.6.2 (Ubuntu 22.04.1 LTS) on a hetzner vserver. Nextcloud version 28.0.5
Any suggestions where to start?

nebulon

Oh I see the datadirectory was adjusted! I have no clue how this works inside nextcloud if this wasn't done from the first startup of the app as nextcloud maintains database records for each file on disk and only support a single data root. So presumably things got out of sync. You have to reach to nextcloud community for such internals I am afraid, I assumed the volume was used via the external storage plugin: https://docs.cloudron.io/apps/nextcloud/#attaching-external-storage

nebulon

Are those regular folders, which you can access fine via the webterminal or filemanager? Are the permissions correct? Can you also check the logs on the postgres service while rescan runs, maybe it runs low on memory?

SolarSimon

The folders are in a remote file system mounted via sshfs. When I open FileManger in Cloudron, I don't see a file tree. So no folders at all. But also don't see problems when operating in nextcloud.
When I open that drive with my SFTP-Software, I can see all folders and all show same rights and owner.

Back at cloudron file manager, when I open a different Volume, which is a different Folder mounted from the same drive, I can see the containing folders.

During "Path not found: /username" there is nothing happening in the postgres logs.
But when "Exception during scan: ..." appears in the terminal, postgress logs shows this:

May 08 10:20:142024-05-08 08:20:14.180 UTC [166] user7826f9bec6084e53af12181aa87f33e2@db7826f9bec6084e53af12181aa87f33e2 ERROR: invalid byte sequence for encoding "UTF8": 0xe4 0x63 0x68
May 08 10:20:142024-05-08 08:20:14.180 UTC [166] user7826f9bec6084e53af12181aa87f33e2@db7826f9bec6084e53af12181aa87f33e2 CONTEXT: unnamed portal parameter $8
May 08 10:20:142024-05-08 08:20:14.180 UTC [166] user7826f9bec6084e53af12181aa87f33e2@db7826f9bec6084e53af12181aa87f33e2 STATEMENT: INSERT INTO "oc_filecache" ("mimepart", "mimetype", "mtime", "size", "etag", "storage_mtime", "permissions", "name", "parent", "checksum", "path_hash", "path", "storage") VALUES($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13)
May 08 10:20:142024-05-08 08:20:14.185 UTC [166] user7826f9bec6084e53af12181aa87f33e2@db7826f9bec6084e53af12181aa87f33e2 ERROR: current transaction is aborted, commands ignored until end of transaction block
May 08 10:20:142024-05-08 08:20:14.185 UTC [166] user7826f9bec6084e53af12181aa87f33e2@db7826f9bec6084e53af12181aa87f33e2 STATEMENT: INSERT INTO "oc_filecache" ("mimepart", "mimetype", "mtime", "size", "etag", "storage_mtime", "permissions", "name", "parent", "checksum", "path_hash", "path", "storage") VALUES($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13)
May 08 10:20:142024-05-08 08:20:14.187 UTC [166] user7826f9bec6084e53af12181aa87f33e2@db7826f9bec6084e53af12181aa87f33e2 ERROR: current transaction is aborted, commands ignored until end of transaction block
May 08 10:20:142024-05-08 08:20:14.187 UTC [166] user7826f9bec6084e53af12181aa87f33e2@db7826f9bec6084e53af12181aa87f33e2 STATEMENT: INSERT INTO "oc_filecache" ("mimepart", "mimetype", "mtime", "size", "etag", "storage_mtime", "permissions", "name", "parent", "checksum", "path_hash", "path", "storage") VALUES($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13)
May 08 10:20:142024-05-08 08:20:14.189 UTC [166] user7826f9bec6084e53af12181aa87f33e2@db7826f9bec6084e53af12181aa87f33e2 ERROR: current transaction is aborted, commands ignored until end of transaction block
May 08 10:20:142024-05-08 08:20:14.189 UTC [166] user7826f9bec6084e53af12181aa87f33e2@db7826f9bec6084e53af12181aa87f33e2 STATEMENT: SELECT "filecache"."fileid", "storage", "path", "path_hash", "filecache"."parent", "filecache"."name", "mimetype", "mimepart", "size", "mtime", "storage_mtime", "encrypted", "etag", "filecache"."permissions", "checksum", "unencrypted_size", "metadata_etag", "creation_time", "upload_time", "meta"."json" AS "meta_json", "meta"."sync_token" AS "meta_sync_token" FROM "oc_filecache" "filecache" LEFT JOIN "oc_filecache_extended" "fe" ON "filecache"."fileid" = "fe"."fileid" LEFT JOIN "oc_files_metadata" "meta" ON "filecache"."fileid" = "meta"."file_id" WHERE ("storage" = $1) AND ("path_hash" = $2)
May 08 10:20:142024-05-08 08:20:14.190 UTC [166] user7826f9bec6084e53af12181aa87f33e2@db7826f9bec6084e53af12181aa87f33e2 ERROR: current transaction is aborted, commands ignored until end of transaction block
May 08 10:20:142024-05-08 08:20:14.190 UTC [166] user7826f9bec6084e53af12181aa87f33e2@db7826f9bec6084e53af12181aa87f33e2 STATEMENT: SELECT "uid", "displayname", "password" FROM "oc_users" WHERE "uid_lower" = $1

Years ago I migrated this nextcloud from a vserver to a nextcloud within cloudron.
Therefore I converted the database from mysql and imported to nextcloud. Maybe relicts?

nebulon

Oh that sshfs adds some interesting angle to the issue. Can you browse the filetree of that sshfs mount via SSH from the server itself without any errors?

If you encounter strange behavior, also check the systemlog for potential kernel errors. Does a remount solve anything in that situation?

SolarSimon

@nebulon said in filescan errors:

Can you browse the filetree

Yep, no problems here

Nextcloud Log while filescan

May 08 14:42:10- - - [08/May/2024:12:42:10 +0000] "GET /index.php/login HTTP/1.1" 200 7156 "-" "Mozilla (CloudronHealth)"
May 08 14:42:1812:M 08 May 2024 12:42:18.008 * 10000 changes in 60 seconds. Saving...
May 08 14:42:1812:M 08 May 2024 12:42:18.012 * Background saving started by pid 368
May 08 14:42:19368:C 08 May 2024 12:42:19.389 * DB saved on disk
May 08 14:42:19368:C 08 May 2024 12:42:19.391 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 1 MB
May 08 14:42:1912:M 08 May 2024 12:42:19.415 * Background saving terminated with success
May 08 14:42:20- - - [08/May/2024:12:42:20 +0000] "GET /index.php/login HTTP/1.1" 200 7151 "-" "Mozilla (CloudronHealth)"
May 08 14:42:30- - - [08/May/2024:12:42:30 +0000] "GET /index.php/login HTTP/1.1" 200 7155 "-" "Mozilla (CloudronHealth)"
May 08 14:42:40- - - [08/May/2024:12:42:40 +0000] "GET /index.php/login HTTP/1.1" 200 7155 "-" "Mozilla (CloudronHealth)"
May 08 14:42:50- - - [08/May/2024:12:42:50 +0000] "GET /index.php/login HTTP/1.1" 200 7149 "-" "Mozilla (CloudronHealth)"
May 08 14:43:00- - - [08/May/2024:12:43:00 +0000] "GET /index.php/login HTTP/1.1" 200 7157 "-" "Mozilla (CloudronHealth)"
May 08 14:43:10- - - [08/May/2024:12:43:10 +0000] "GET /index.php/login HTTP/1.1" 200 7155 "-" "Mozilla (CloudronHealth)"
May 08 14:43:20- - - [08/May/2024:12:43:20 +0000] "GET /index.php/login HTTP/1.1" 200 7153 "-" "Mozilla (CloudronHealth)"
May 08 14:43:30- - - [08/May/2024:12:43:30 +0000] "GET /index.php/login HTTP/1.1" 200 7154 "-" "Mozilla (CloudronHealth)"
May 08 14:43:40- - - [08/May/2024:12:43:40 +0000] "GET /index.php/login HTTP/1.1" 200 7155 "-" "Mozilla (CloudronHealth)"

Cloudron Box Log while filescan

May 08 14:42:10box:apphealthmonitor app health: 7 running / 3 stopped / 0 unresponsive
...

Via Cloudron File Manager I opend Nextclouds config.php. At the same time the system log shows this

May 08 14:51:23box:server no such route: GET /api/v1/apps/7826f9be-c608-4e53-af12-181aa87f33e2/files/config/config.phpfiles/app-7826f9be-c608-4e53-af12-181aa87f33e2/config%2Fconfig.php?access_token=<redacted>
[ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent to the client
at new NodeError (node:internal/errors:399:5)
at ServerResponse.setHeader (node:_http_outgoing:645:11)
at ServerResponse.header (/home/yellowtent/box/node_modules/express/lib/response.js:794:10)
at ServerResponse.json (/home/yellowtent/box/node_modules/express/lib/response.js:275:10)
at ServerResponse.send (/home/yellowtent/box/node_modules/express/lib/response.js:162:21)
at /home/yellowtent/box/node_modules/connect-lastmile/lib/index.js:78:28
at Layer.handle_error (/home/yellowtent/box/node_modules/express/lib/router/layer.js:71:5)
at trim_prefix (/home/yellowtent/box/node_modules/express/lib/router/index.js:326:13)
at /home/yellowtent/box/node_modules/express/lib/router/index.js:286:9
at Function.process_params (/home/yellowtent/box/node_modules/express/lib/router/index.js:346:12)

nebulon

Strange the URL clearly has the file/api path totally wrong: /api/v1/apps/7826f9be-c608-4e53-af12-181aa87f33e2/files/config/config.phpfiles/app-7826f9be-c608-4e53-af12-181aa87f33e2/config%2Fconfig.php

Not sure where this is coming from to be honest. All in all it appears there are a bunch of probably unrelated issue actually.

So for nextcloud, you can confirm that using the app itself and downloading/uploading files works just fine, only the file rescanning via occ fails? This may or may not be related to sshfs potentially being slow. Not too sure how to debug if ther are no system errors (journalctl --system or dmesg). Debugging Nextcloud's php code is a bit out of scope for us.

The Cloudron filemanager issue should be fixed though if this is a real problem. Do you have other apps which reside in that sshfs mount point and does the filemanger work there?

SolarSimon

@nebulon said in filescan errors:

So for nextcloud, you can confirm that using the app itself and downloading/uploading files works just fine, only the file rescanning via occ fails?

yep. I had some issues with umlauts when I migrated. But since long there are no file errors while operating.

@nebulon said in filescan errors:

Do you have other apps which reside in that sshfs mount point and does the filemanger work there?

yes, there is another nextcloud app running on a different domain. Dont have files:scan errors there and I see files via FileManager.

@nebulon said in filescan errors:

may or may not be related to sshfs potentially being slow.

I don't see issues there. nextcloud is working fine.

nebulon

To take a step back and remove the Cloudron filemanager out of the equation, if you can browse the filesystem via the webterminal into that app instance, does it look correct? Are the folders for those usernames there?

SolarSimon

@nebulon said in filescan errors:

if you can browse the filesystem via the webterminal into that app instance, does it look correct?

when I go there, I see the local Data Directory of the app, but not the mounted volume content.
This applies for both instance of Nextcloud

When I navigate to /media/volume in the app Terminal (not File Manager), I can see the users Folders

nebulon

that is to be expected then, the filemanager only shows the mointpoint of /app/data into the app. How is /media/volume used within Nextcloud then though?

SolarSimon

I don't excactly get your question. Maybe this screen is the answer?

'datadirectory' => '/media/nextcloudbox',

nebulon

Oh I see the datadirectory was adjusted! I have no clue how this works inside nextcloud if this wasn't done from the first startup of the app as nextcloud maintains database records for each file on disk and only support a single data root. So presumably things got out of sync. You have to reach to nextcloud community for such internals I am afraid, I assumed the volume was used via the external storage plugin: https://docs.cloudron.io/apps/nextcloud/#attaching-external-storage

Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.

Cloudron Forum

filescan errors