Nextcloud Migration Error

creative567145

Hi

Have tried 10+ times to run a Nextcloud migration.

But we are constantly getting the same error.

We hope that this can be solved. Since this forum is public, the solution will be available for everyone else who will have the same problem

Task Error in the Nextcloud > Repair Tab:

Addons Error: Unexpected response code or HTTP error when piping /home/yellowtent/appsdata/57af8641-8ca3-4a61-b18a-cbf6308fb45e/postgresqldump to http://172.18.30.2:3000/databases/db57af86418ca34a61b18acbf6308fb45e/restore?access_token=072212caff8830b05a63b50516fa15953eeb2cd970e6668dbf7268d8dd43d609dcb58e6bf039e0b3befdd38b137de76828624185d3b0d6340a38f4e7aa467a2842bd64880a0392f4644e69faf49c4049fedcb4a2b3b41f7934a82d91e2453ea7172dad77131adb9389793dad635596d1c8db452a816a70985ed2941bfdc5d78e&username=user57af86418ca34a61b18acbf6308fb45e: status 500 complete false

Logs:

17 06:41:29 box:tasks update 2805: {"percent":65,"message":"Download finished 2024-07-15-190000-564/app_vtmg.link_v4.22.2/postgresqldump to /home/yellowtent/appsdata/57af8641-8ca3-4a61-b18a-cbf6308fb45e/postgresqldump"}
Jul 17 06:41:29 box:backupformat/rsync Recreating empty directories in {"localRoot":"/home/yellowtent/appsdata/57af8641-8ca3-4a61-b18a-cbf6308fb45e","layout":[]}

Jul 17 06:44:37 box:backuptask downloadApp: time: 27101.091

Jul 17 06:44:37 box:tasks update 2805: {"percent":70,"message":"Restoring addons"}

Jul 17 06:44:37 box:services restoreAddons

Jul 17 06:44:37 box:services restoreAddons: restoring ["postgresql","sendmail","ldap","redis","localstorage","scheduler","turn"]

Jul 17 06:44:37 box:services Restore postgresql

Jul 17 06:44:37 box:services pipeFileToRequest: piping /home/yellowtent/appsdata/57af8641-8ca3-4a61-b18a-cbf6308fb45e/postgresqldump to http://172.18.30.2:3000/databases/db57af86418ca34a61b18acbf6308fb45e/restore?access_token=072212caff8830b05a63b50516fa15953eeb2cd970e6668dbf7268d8dd43d609dcb58e6bf039e0b3befdd38b137de76828624185d3b0d6340a38f4e7aa467a2842bd64880a0392f4644e69faf49c4049fedcb4a2b3b41f7934a82d91e2453ea7172dad77131adb9389793dad635596d1c8db452a816a70985ed2941bfdc5d78e&username=user57af86418ca34a61b18acbf6308fb45e

Jul 17 06:48:43 box:services pipeFileToRequest: piped /home/yellowtent/appsdata/57af8641-8ca3-4a61-b18a-cbf6308fb45e/postgresqldump to http://172.18.30.2:3000/databases/db57af86418ca34a61b18acbf6308fb45e/restore?access_token=072212caff8830b05a63b50516fa15953eeb2cd970e6668dbf7268d8dd43d609dcb58e6bf039e0b3befdd38b137de76828624185d3b0d6340a38f4e7aa467a2842bd64880a0392f4644e69faf49c4049fedcb4a2b3b41f7934a82d91e2453ea7172dad77131adb9389793dad635596d1c8db452a816a70985ed2941bfdc5d78e&username=user57af86418ca34a61b18acbf6308fb45e

Jul 17 06:51:44 box:services pipeFileToRequest: request completed with status code 500

Jul 17 06:51:44 box:apptask run: app error for state pending_restore: BoxError: Unexpected response code or HTTP error when piping /home/yellowtent/appsdata/57af8641-8ca3-4a61-b18a-cbf6308fb45e/postgresqldump to http://172.18.30.2:3000/databases/db57af86418ca34a61b18acbf6308fb45e/restore?access_token=072212caff8830b05a63b50516fa15953eeb2cd970e6668dbf7268d8dd43d609dcb58e6bf039e0b3befdd38b137de76828624185d3b0d6340a38f4e7aa467a2842bd64880a0392f4644e69faf49c4049fedcb4a2b3b41f7934a82d91e2453ea7172dad77131adb9389793dad635596d1c8db452a816a70985ed2941bfdc5d78e&username=user57af86418ca34a61b18acbf6308fb45e: status 500 complete false at ClientRequest.<anonymous> (/home/yellowtent/box/src/services.js:1311:60) at ClientRequest.emit (node:events:518:28) at ClientRequest.emit (node:domain:488:12) at HTTPParser.parserOnIncomingClient (node:_http_client:698:27) at HTTPParser.parserOnHeadersComplete (node:_http_common:119:17) at Socket.socketOnData (node:_http_client:540:22) at Socket.emit (node:events:518:28) at Socket.emit (node:domain:488:12) at addChunk (node:internal/streams/readable:559:12) at readableAddChunkPushByteMode (node:internal/streams/readable:510:3) { reason: 'Addons Error', details: {} }

Jul 17 06:51:44 box:taskworker Task took 27570.192 seconds

Jul 17 06:51:44 box:tasks setCompleted - 2805: {"result":null,"error":{"stack":"BoxError: Unexpected response code or HTTP error when piping /home/yellowtent/appsdata/57af8641-8ca3-4a61-b18a-cbf6308fb45e/postgresqldump to http://172.18.30.2:3000/databases/db57af86418ca34a61b18acbf6308fb45e/restore?access_token=072212caff8830b05a63b50516fa15953eeb2cd970e6668dbf7268d8dd43d609dcb58e6bf039e0b3befdd38b137de76828624185d3b0d6340a38f4e7aa467a2842bd64880a0392f4644e69faf49c4049fedcb4a2b3b41f7934a82d91e2453ea7172dad77131adb9389793dad635596d1c8db452a816a70985ed2941bfdc5d78e&username=user57af86418ca34a61b18acbf6308fb45e: status 500 complete false\n at ClientRequest.<anonymous> (/home/yellowtent/box/src/services.js:1311:60)\n at ClientRequest.emit (node:events:518:28)\n at ClientRequest.emit (node:domain:488:12)\n at HTTPParser.parserOnIncomingClient (node:_http_client:698:27)\n at HTTPParser.parserOnHeadersComplete (node:_http_common:119:17)\n at Socket.socketOnData (node:_http_client:540:22)\n at Socket.emit (node:events:518:28)\n at Socket.emit (node:domain:488:12)\n at addChunk (node:internal/streams/readable:559:12)\n at readableAddChunkPushByteMode (node:internal/streams/readable:510:3)","name":"BoxError","reason":"Addons Error","details":{},"message":"Unexpected response code or HTTP error when piping /home/yellowtent/appsdata/57af8641-8ca3-4a61-b18a-cbf6308fb45e/postgresqldump to http://172.18.30.2:3000/databases/db57af86418ca34a61b18acbf6308fb45e/restore?access_token=072212caff8830b05a63b50516fa15953eeb2cd970e6668dbf7268d8dd43d609dcb58e6bf039e0b3befdd38b137de76828624185d3b0d6340a38f4e7aa467a2842bd64880a0392f4644e69faf49c4049fedcb4a2b3b41f7934a82d91e2453ea7172dad77131adb9389793dad635596d1c8db452a816a70985ed2941bfdc5d78e&username=user57af86418ca34a61b18acbf6308fb45e: status 500 complete false"}}

Jul 17 06:51:44 box:tasks update 2805: {"percent":100,"result":null,"error":{"stack":"BoxError: Unexpected response code or HTTP error when piping /home/yellowtent/appsdata/57af8641-8ca3-4a61-b18a-cbf6308fb45e/postgresqldump to http://172.18.30.2:3000/databases/db57af86418ca34a61b18acbf6308fb45e/restore?access_token=072212caff8830b05a63b50516fa15953eeb2cd970e6668dbf7268d8dd43d609dcb58e6bf039e0b3befdd38b137de76828624185d3b0d6340a38f4e7aa467a2842bd64880a0392f4644e69faf49c4049fedcb4a2b3b41f7934a82d91e2453ea7172dad77131adb9389793dad635596d1c8db452a816a70985ed2941bfdc5d78e&username=user57af86418ca34a61b18acbf6308fb45e: status 500 complete false\n at ClientRequest.<anonymous> (/home/yellowtent/box/src/services.js:1311:60)\n at ClientRequest.emit (node:events:518:28)\n at ClientRequest.emit (node:domain:488:12)\n at HTTPParser.parserOnIncomingClient (node:_http_client:698:27)\n at HTTPParser.parserOnHeadersComplete (node:_http_common:119:17)\n at Socket.socketOnData (node:_http_client:540:22)\n at Socket.emit (node:events:518:28)\n at Socket.emit (node:domain:488:12)\n at addChunk (node:internal/streams/readable:559:12)\n at readableAddChunkPushByteMode (node:internal/streams/readable:510:3)","name":"BoxError","reason":"Addons Error","details":{},"message":"Unexpected response code or HTTP error when piping /home/yellowtent/appsdata/57af8641-8ca3-4a61-b18a-cbf6308fb45e/postgresqldump to http://172.18.30.2:3000/databases/db57af86418ca34a61b18acbf6308fb45e/restore?access_token=072212caff8830b05a63b50516fa15953eeb2cd970e6668dbf7268d8dd43d609dcb58e6bf039e0b3befdd38b137de76828624185d3b0d6340a38f4e7aa467a2842bd64880a0392f4644e69faf49c4049fedcb4a2b3b41f7934a82d91e2453ea7172dad77131adb9389793dad635596d1c8db452a816a70985ed2941bfdc5d78e&username=user57af86418ca34a61b18acbf6308fb45e: status 500 complete false"}}
[no timestamp]  Unexpected response code or HTTP error when piping /home/yellowtent/appsdata/57af8641-8ca3-4a61-b18a-cbf6308fb45e/postgresqldump to http://172.18.30.2:3000/databases/db57af86418ca34a61b18acbf6308fb45e/restore?access_token=072212caff8830b05a63b50516fa15953eeb2cd970e6668dbf7268d8dd43d609dcb58e6bf039e0b3befdd38b137de76828624185d3b0d6340a38f4e7aa467a2842bd64880a0392f4644e69faf49c4049fedcb4a2b3b41f7934a82d91e2453ea7172dad77131adb9389793dad635596d1c8db452a816a70985ed2941bfdc5d78e&username=user57af86418ca34a61b18acbf6308fb45e: status 500 complete false

Additional info if needed:

The migration is run from a Linode VPS (Nexcloud on Cloudron) Ubuntu 22.04 LTS OS
to a Dedicated server Ubuntu 24.04 LTS OS
In order so that there isn't any PostgreSQL database corruption, before the Backup was created we put Nextcloud in Recovery mode (so that the database isn't edited). We also tested without putting it in Recovery mode and the result was the same.
The migration is running from a Linode Object Storage

nebulon

Looks like this is related to the postgres database service. Can you also check the logs of that one? Depending on the dump size, it may also help to increase the memory limit of that service.

joseph

Services -> PostgreSQL -> Bump the memory limit

creative567145

Thank you @nebulon and @joseph on your help

The PostreSQL service was already set to the max available 64GB RAM (from total 132GB RAM)

@nebulon here is the requested log: (thank you again)

Jul 17 06:51:43 2024-07-17 04:51:43.846 UTC [1062] ERROR: temporary file size exceeds temp_file_limit (1048576kB)

Jul 17 06:51:43 2024-07-17 04:51:43.846 UTC [1062] STATEMENT: CREATE INDEX oc_notifications_object ON public.oc_notifications USING btree (object_type, object_id);

Jul 17 06:51:43 2024-07-17 04:51:43.848 UTC [1007] root@db57af86418ca34a61b18acbf6308fb45e CONTEXT: parallel worker

Jul 17 06:51:43 2024-07-17 04:51:43.848 UTC [1007] root@db57af86418ca34a61b18acbf6308fb45e ERROR: temporary file size exceeds temp_file_limit (1048576kB)

Jul 17 06:51:43 2024-07-17 04:51:43.848 UTC [1007] root@db57af86418ca34a61b18acbf6308fb45e STATEMENT: CREATE INDEX oc_notifications_object ON public.oc_notifications USING btree (object_type, object_id);

Jul 17 06:51:43 2024-07-17 04:51:43.851 UTC [49] LOG: background worker "parallel worker" (PID 1062) exited with exit code 1

Jul 17 06:51:44 CONTEXT: parallel worker

Jul 17 06:51:44 restore: failed to restore database. code=3

Jul 17 06:51:44 restore: stderr from db import: ERROR: temporary file size exceeds temp_file_limit (1048576kB)

Jul 17 22:20:03 [GET] /healthcheck

Jul 17 22:20:03 healthcheck: disconnected

The PostgreSQL Dump file is 15GB.
From what we understand here, the limit is 1GB.
How can we increase this limit or solve this problem another way?

nebulon

That is some large Nextcloud database instance! So I think the issue is with the postgres service creating a temp file in /tmp within the container. This is a transient volume, can you check if the disk it is created at has enough space temporarily? You can find the actual system path via docker inspect postgresql looking for the Source in the /tmp Mount.

I think we will have to rewrite that part to not rely on tmp files in the future.

girish

This might be a regression I introduced in Cloudron 8. @creative567145 are you already on Cloudron 8 ?

Assuming, you are on Cloudron 8, here's a workaround. A bit rough but should work:

SSH into server and docker exec -ti postgresql /bin/bash
- Edit the file /var/lib/postgresql/14/main/postgresql.conf. There is a field there temp_file_limit = 1GB . Remove that line altogether (the default is unlimited).
- supervisorctl restart postgresql

Now try the import.

girish

That change came from https://forum.cloudron.io/topic/11781/customize-postgresql-settings-limits

creative567145

Thank you @nebulon and @girish very much!

Yes @nebulon it's a quite big database. This Nextcloud has around 6000 users.
The disk has above 1TB SSD so that was ok.

Thank you @girish for the detailed steps. They were very helpful!
The import was a success.

Note: The change isn't persistent on a Reboot.
It would be nice to change this:
max_connections = 500
to this:
max_connections = 3000
If the server is powerful enough, it should be able to sustain the x6 boost.
The only challenge is how to make the change persistent after Reboot

girish

OK. I have added a change in 8.0.2 where you can add a custom.conf along side the postgres configuration . It's not backed up but atleast it is persistent across restarts and updates.

creative567145

Thank you @girish that will be very useful

girish

@creative567145 you can update to 8.0.2 and try the custom configuration . It's not documented but if this is requested more often, I can put this in the docs.

creative567145

I wish that we could do that, but this specific client doesn't want the configuration to be done. He is afraid that something might break. We assured him that all will be ok, but he insists. We must respect his wishes.

Perhaps in the near future we will try it for some other client with high number of users.

Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.

Cloudron Forum

Nextcloud Migration Error