Error while recovering backup

[andreasdueren]

From the logs:

Jan 25 21:51:05box:tasks update 10014: {"percent":65,"message":"Download finished 2024-01-26-034812-737/app_domain.tld_v1.37.3/data/files/fc238c09-4bf6-4515-9404-36caa6fc7ac0__641a888e5b2b754dabb97b20cb3fff990eeff041.jpeg to /home/yellowtent/appsdata/4114dc19-cacb-4d4d-aa88-2f54578ec522/data/files/fc238c09-4bf6-4515-9404-36caa6fc7ac0__641a888e5b2b754dabb97b20cb3fff990eeff041.jpeg"}
to /home/yellowtent/appsdata/4114dc19-cacb-4d4d-aa88-2f54578ec522/data/files/f9709240-bacd-44b5-848c-b23fe0fce416__f72164e261a70c44dc69373e74c2f82d3bbf2a23.jpeg"}
Jan 25 21:51:06box:tasks update 10014: {"percent":65,"message":"Download finished 2024-01-26-034812-737/app_domain.tld_v1.37.3/postgresqldump to /home/yellowtent/appsdata/4114dc19-cacb-4d4d-aa88-2f54578ec522/postgresqldump"}
Jan 25 21:51:06box:backupformat/rsync Recreating empty directories in {"localRoot":"/home/yellowtent/appsdata/4114dc19-cacb-4d4d-aa88-2f54578ec522","layout":[]}
Jan 25 21:51:06box:apptask run: app error for state pending_restore: BoxError: unable to chmod: ENOENT: no such file or directory, chmod '/home/yellowtent/appsdata/4114dc19-cacb-4d4d-aa88-2f54578ec522/data/extensions/axios/lib/adapters/http.js' at restoreFsMetadata (/home/yellowtent/box/src/backupformat/rsync.js:149:31) at async Object.download (/home/yellowtent/box/src/backupformat/rsync.js:239:5) at async download (/home/yellowtent/box/src/backuptask.js:100:5) at async Object.downloadApp (/home/yellowtent/box/src/backuptask.js:133:5) at async install (/home/yellowtent/box/src/apptask.js:335:9) { reason: 'FileSystem Error', details: {} }
Jan 25 21:51:06box:tasks setCompleted - 10014: {"result":null,"error":{"stack":"BoxError: unable to chmod: ENOENT: no such file or directory, chmod '/home/yellowtent/appsdata/4114dc19-cacb-4d4d-aa88-2f54578ec522/data/extensions/axios/lib/adapters/http.js'\n at restoreFsMetadata (/home/yellowtent/box/src/backupformat/rsync.js:149:31)\n at async Object.download (/home/yellowtent/box/src/backupformat/rsync.js:239:5)\n at async download (/home/yellowtent/box/src/backuptask.js:100:5)\n at async Object.downloadApp (/home/yellowtent/box/src/backuptask.js:133:5)\n at async install (/home/yellowtent/box/src/apptask.js:335:9)","name":"BoxError","reason":"FileSystem Error","details":{},"message":"unable to chmod: ENOENT: no such file or directory, chmod '/home/yellowtent/appsdata/4114dc19-cacb-4d4d-aa88-2f54578ec522/data/extensions/axios/lib/adapters/http.js'"}}
Jan 25 21:51:06box:tasks update 10014: {"percent":100,"result":null,"error":{"stack":"BoxError: unable to chmod: ENOENT: no such file or directory, chmod '/home/yellowtent/appsdata/4114dc19-cacb-4d4d-aa88-2f54578ec522/data/extensions/axios/lib/adapters/http.js'\n at restoreFsMetadata (/home/yellowtent/box/src/backupformat/rsync.js:149:31)\n at async Object.download (/home/yellowtent/box/src/backupformat/rsync.js:239:5)\n at async download (/home/yellowtent/box/src/backuptask.js:100:5)\n at async Object.downloadApp (/home/yellowtent/box/src/backuptask.js:133:5)\n at async install (/home/yellowtent/box/src/apptask.js:335:9)","name":"BoxError","reason":"FileSystem Error","details":{},"message":"unable to chmod: ENOENT: no such file or directory, chmod '/home/yellowtent/appsdata/4114dc19-cacb-4d4d-aa88-2f54578ec522/data/extensions/axios/lib/adapters/http.js'"}}
Jan 25 21:51:06box:taskworker Task took 26.451 seconds
unable to chmod: ENOENT: no such file or directory, chmod '/home/yellowtent/appsdata/4114dc19-cacb-4d4d-aa88-2f54578ec522/data/extensions/axios/lib/adapters/http.js'

[andreasdueren]

After reboot of the server I now got this:

Jan 25 22:06:51box:docker pullImage: {"status":"Pull complete","progressDetail":{},"id":"0a4ddf65a667"}
Jan 25 22:06:51box:docker pullImage: {"status":"Digest: sha256:af0794d91424da822ab874b1ddaf6a06ce43edba0604cd2f075c43cf9dd74cc2"}
Jan 25 22:06:51box:docker pullImage: {"status":"Status: Downloaded newer image for cloudron/io.directus9.cloudronapp:20231222-105753-000f6a67c"}
Jan 25 22:06:51box:docker downloaded image cloudron/io.directus9.cloudronapp:20231222-105753-000f6a67c . error: false
Jan 25 22:06:51box:tasks update 10018: {"percent":50,"message":"Creating app data directory"}
Jan 25 22:06:51box:tasks update 10018: {"percent":65,"message":"Downloading backup and restoring addons"}
Jan 25 22:06:51box:services setupAddons: Setting up ["localstorage","postgresql","redis","ldap","sendmail"]
Jan 25 22:06:51box:services setupAddons: setting up addon localstorage with options {}
Jan 25 22:06:51box:services setupLocalStorage
Jan 25 22:06:51box:shell createVolume spawn: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/mkdirvolume.sh /home/yellowtent/appsdata/4114dc19-cacb-4d4d-aa88-2f54578ec522/data
Jan 25 22:06:51box:services Setting up postgresql
Jan 25 22:06:51box:services setupAddons: setting up addon postgresql with options {}
Jan 25 22:06:51box:apptask run: app error for state pending_restore: BoxError: Network error setting up postgresql: connect ECONNREFUSED 172.18.30.2:3000 at Object.setupPostgreSql [as setup] (/home/yellowtent/box/src/services.js:1408:29) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async Object.setupAddons (/home/yellowtent/box/src/services.js:604:9) at async install (/home/yellowtent/box/src/apptask.js:333:9) { reason: 'Addons Error', details: {} }
Jan 25 22:06:51box:tasks setCompleted - 10018: {"result":null,"error":{"stack":"BoxError: Network error setting up postgresql: connect ECONNREFUSED 172.18.30.2:3000\n at Object.setupPostgreSql [as setup] (/home/yellowtent/box/src/services.js:1408:29)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async Object.setupAddons (/home/yellowtent/box/src/services.js:604:9)\n at async install (/home/yellowtent/box/src/apptask.js:333:9)","name":"BoxError","reason":"Addons Error","details":{},"message":"Network error setting up postgresql: connect ECONNREFUSED 172.18.30.2:3000"}}
Jan 25 22:06:51box:tasks update 10018: {"percent":100,"result":null,"error":{"stack":"BoxError: Network error setting up postgresql: connect ECONNREFUSED 172.18.30.2:3000\n at Object.setupPostgreSql [as setup] (/home/yellowtent/box/src/services.js:1408:29)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async Object.setupAddons (/home/yellowtent/box/src/services.js:604:9)\n at async install (/home/yellowtent/box/src/apptask.js:333:9)","name":"BoxError","reason":"Addons Error","details":{},"message":"Network error setting up postgresql: connect ECONNREFUSED 172.18.30.2:3000"}}
Jan 25 22:06:51box:taskworker Task took 10.064 seconds
Network error setting up postgresql: connect ECONNREFUSED 172.18.30.2:3000

[andreasdueren]

This is Directus 1.37.3. I tried rolling back to various backups but the earliest one that actually works is 8 versions away on Monday which is anything but optimal. Any Ideas? @girish

[girish]

@andreasdueren said in Error while recovering backup:

Jan 25 22:06:51box:apptask run: app error for state pending_restore: BoxError: Network error setting up postgresql: connect ECONNREFUSED 172.18.30.2:3000 at Object.setupPostgreSql [as setup] (/home/yellowtent/box/src/services.js:1408:29) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async Object.setupAddons (/home/yellowtent/box/src/services.js:604:9) at async install (/home/yellowtent/box/src/apptask.js:333:9) { reason: 'Addons Error', details: {} }
Jan 25 22:06:51box:tasks setCompleted - 10018: {"result":null,"error":{"stack":"BoxError: Network error setting up postgresql: connect ECONNREFUSED 172.18.30.2:3000\n at Object.setupPostgreSql [as setup] (/home/yellowtent/box/src/services.js:1408:29)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async Object.setupAddons (/home/yellowtent/box/src/services.js:604:9)\n at async install (/home/yellowtent/box/src/apptask.js:333:9)","name":"BoxError","reason":"Addons Error","details":{},"message":"Network error setting up postgresql: connect ECONNREFUSED 172.18.30.2:3000"}}
Jan 25 22:06:51box:tasks update 10018: {"percent":100,"result":null,"error":{"stack":"BoxError: Network error setting up postgresql: connect ECONNREFUSED 172.18.30.2:3000\n at Object.setupPostgreSql [as setup] (/home/yellowtent/box/src/services.js:1408:29)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async Object.setupAddons (/home/yellowtent/box/src/services.js:604:9)\n at async install (/home/yellowtent/box/src/apptask.js:333:9)","name":"BoxError","reason":"Addons Error","details":{},"message":"Network error setting up postgresql: connect ECONNREFUSED 172.18.30.2:3000"}}

When the above happens it's because postgresql is not running or does not have enough memory. Can you go to Services and check if PostgreSQL is running? If it is, give it a lot more memory and then attempt the restore again. Usually, the restore takes more memory in databases. You can always reduce it after restoring.

[andreasdueren]

@girish I had checked that and given 12GB of memory. Bumped it to 64 now but still not working. prostreSQL still running

[andreasdueren]

@andreasdueren said in Error while recovering backup:

From the logs:

Jan 25 21:51:05box:tasks update 10014: {"percent":65,"message":"Download finished 2024-01-26-034812-737/app_domain.tld_v1.37.3/data/files/fc238c09-4bf6-4515-9404-36caa6fc7ac0__641a888e5b2b754dabb97b20cb3fff990eeff041.jpeg to /home/yellowtent/appsdata/4114dc19-cacb-4d4d-aa88-2f54578ec522/data/files/fc238c09-4bf6-4515-9404-36caa6fc7ac0__641a888e5b2b754dabb97b20cb3fff990eeff041.jpeg"}
to /home/yellowtent/appsdata/4114dc19-cacb-4d4d-aa88-2f54578ec522/data/files/f9709240-bacd-44b5-848c-b23fe0fce416__f72164e261a70c44dc69373e74c2f82d3bbf2a23.jpeg"}
Jan 25 21:51:06box:tasks update 10014: {"percent":65,"message":"Download finished 2024-01-26-034812-737/app_domain.tld_v1.37.3/postgresqldump to /home/yellowtent/appsdata/4114dc19-cacb-4d4d-aa88-2f54578ec522/postgresqldump"}
Jan 25 21:51:06box:backupformat/rsync Recreating empty directories in {"localRoot":"/home/yellowtent/appsdata/4114dc19-cacb-4d4d-aa88-2f54578ec522","layout":[]}
Jan 25 21:51:06box:apptask run: app error for state pending_restore: BoxError: unable to chmod: ENOENT: no such file or directory, chmod '/home/yellowtent/appsdata/4114dc19-cacb-4d4d-aa88-2f54578ec522/data/extensions/axios/lib/adapters/http.js' at restoreFsMetadata (/home/yellowtent/box/src/backupformat/rsync.js:149:31) at async Object.download (/home/yellowtent/box/src/backupformat/rsync.js:239:5) at async download (/home/yellowtent/box/src/backuptask.js:100:5) at async Object.downloadApp (/home/yellowtent/box/src/backuptask.js:133:5) at async install (/home/yellowtent/box/src/apptask.js:335:9) { reason: 'FileSystem Error', details: {} }
Jan 25 21:51:06box:tasks setCompleted - 10014: {"result":null,"error":{"stack":"BoxError: unable to chmod: ENOENT: no such file or directory, chmod '/home/yellowtent/appsdata/4114dc19-cacb-4d4d-aa88-2f54578ec522/data/extensions/axios/lib/adapters/http.js'\n at restoreFsMetadata (/home/yellowtent/box/src/backupformat/rsync.js:149:31)\n at async Object.download (/home/yellowtent/box/src/backupformat/rsync.js:239:5)\n at async download (/home/yellowtent/box/src/backuptask.js:100:5)\n at async Object.downloadApp (/home/yellowtent/box/src/backuptask.js:133:5)\n at async install (/home/yellowtent/box/src/apptask.js:335:9)","name":"BoxError","reason":"FileSystem Error","details":{},"message":"unable to chmod: ENOENT: no such file or directory, chmod '/home/yellowtent/appsdata/4114dc19-cacb-4d4d-aa88-2f54578ec522/data/extensions/axios/lib/adapters/http.js'"}}
Jan 25 21:51:06box:tasks update 10014: {"percent":100,"result":null,"error":{"stack":"BoxError: unable to chmod: ENOENT: no such file or directory, chmod '/home/yellowtent/appsdata/4114dc19-cacb-4d4d-aa88-2f54578ec522/data/extensions/axios/lib/adapters/http.js'\n at restoreFsMetadata (/home/yellowtent/box/src/backupformat/rsync.js:149:31)\n at async Object.download (/home/yellowtent/box/src/backupformat/rsync.js:239:5)\n at async download (/home/yellowtent/box/src/backuptask.js:100:5)\n at async Object.downloadApp (/home/yellowtent/box/src/backuptask.js:133:5)\n at async install (/home/yellowtent/box/src/apptask.js:335:9)","name":"BoxError","reason":"FileSystem Error","details":{},"message":"unable to chmod: ENOENT: no such file or directory, chmod '/home/yellowtent/appsdata/4114dc19-cacb-4d4d-aa88-2f54578ec522/data/extensions/axios/lib/adapters/http.js'"}}
Jan 25 21:51:06box:taskworker Task took 26.451 seconds
unable to chmod: ENOENT: no such file or directory, chmod '/home/yellowtent/appsdata/4114dc19-cacb-4d4d-aa88-2f54578ec522/data/extensions/axios/lib/adapters/http.js'

This one seems to be the actual problem with postgreSQL maybe just being a temporary glitch

[andreasdueren]

Is there a way to download only the PostgeSQL Part to get at least most of the data?

[andreasdueren]

Trying to decrypt the backup locally throws this error:

/usr/local/lib/node_modules/cloudron/src/backup-tools.js:261
            const { error, decryptedFilePath } = decryptFilePath(encryptedFilePath, encryption);
                    ^

TypeError: Cannot destructure property 'error' of 'decryptFilePath(...)' as it is null.
    at /usr/local/lib/node_modules/cloudron/src/backup-tools.js:261:21
    at /usr/local/lib/node_modules/cloudron/node_modules/async/dist/async.js:2590:44
    at replenish (/usr/local/lib/node_modules/cloudron/node_modules/async/dist/async.js:448:21)
    at /usr/local/lib/node_modules/cloudron/node_modules/async/dist/async.js:453:13
    at eachLimit (/usr/local/lib/node_modules/cloudron/node_modules/async/dist/async.js:2720:36)
    at awaitable (/usr/local/lib/node_modules/cloudron/node_modules/async/dist/async.js:212:32)
    at Object.eachSeries (/usr/local/lib/node_modules/cloudron/node_modules/async/dist/async.js:2748:16)
    at Object.awaitable (/usr/local/lib/node_modules/cloudron/node_modules/async/dist/async.js:212:32)
    at iteratee (/usr/local/lib/node_modules/cloudron/src/backup-tools.js:252:15)
    at check (/usr/local/lib/node_modules/cloudron/node_modules/async/dist/async.js:5675:13)

[girish]

@andreasdueren the database dump should be in /home/yellowtent/appsdata/<appid>/postgresqldump . It will already be unencrypted.

The CLI tool decryption hasn't been tested in a while, I can look into it.

[andreasdueren]

@girish said in Error while recovering backup:

@andreasdueren the database dump should be in /home/yellowtent/appsdata/<appid>/postgresqldump . It will already be unencrypted.

The CLI tool decryption hasn't been tested in a while, I can look into it.

Can you think of any other way to decrypt it? Manually adding the information into the backup recovery in the ui also won't work.

[andreasdueren]

@girish said in Error while recovering backup:

@andreasdueren the database dump should be in /home/yellowtent/appsdata/<appid>/postgresqldump . It will already be unencrypted.

If there was one, it is now lost since I tried to roll back to the old version.

[andreasdueren]

Update: I think I was able to get the backup back by using the CLI to roll back. Have to check everything still though.

[andreasdueren]

@andreasdueren said in Error while recovering backup:

Update: I think I was able to get the backup back by using the CLI to roll back. Have to check everything still though.

Never mind turned out it didn’t select the backup I wanted and failed the same way.

[girish]

@andreasdueren Can you check if this file is inside wasabi - extensions/axios/lib/adapters/http.js ?

[andreasdueren]

@girish said in Error while recovering backup:

@andreasdueren Can you check if this file is inside wasabi - extensions/axios/lib/adapters/http.js ?

Unfortunately, the backup is encrypted. There is a file, but I do not know if it's corrupt or intact.

[girish]

@andreasdueren ah true, I will fix the cli script

[andreasdueren]

This post is deleted!

[andreasdueren]

This post is deleted!

[girish]

CLI crash is fixed in CLI version 5.4.2 .

[andreasdueren]

Different issues but also related to the decrypting backup: I assume "Error: Invalid initialization vector" is referring to the wrong passphrase?