backup error: "undefined is not mounted"

3246

Even though the restore was completed successfully and the NFS share is mounted through Cloudron OK, I now get the following error when backing back up to it:

"undefined is not mounted"

Feb 23 09:40:14 box:shell backup-snapshot/app_4d8e74a4-5edb-4e30-b344-9c88a36836be (stdout): 2022-02-23T09:40:13.999Z box:backupupload upload completed. error: BoxError: undefined is not mounted
at Object.checkPreconditions (/home/yellowtent/box/src/storage/filesystem.js:86:73)
at processTicksAndRejections (internal/process/task_queues.js:95:5) {
reason: 'FileSystem Error',
details: {}
Feb 23 09:40:14 box:backuptask runBackupUpload: result - {"result":"undefined is not mounted"}
Feb 23 09:40:14 box:shell backup-snapshot/app_4d8e74a4-5edb-4e30-b344-9c88a36836be code: 50, signal: null
Feb 23 09:40:14 box:taskworker Task took 0.613 seconds
Feb 23 09:40:14 box:tasks setCompleted - 991: {"result":null,"error":{"stack":"BoxError: undefined is not mounted\n at /home/yellowtent/box/src/backuptask.js:709:29\n at f (/home/yellowtent/box/node_modules/once/once.js:25:25)\n at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:81:9)\n at ChildProcess.emit (events.js:400:28)\n at Process.ChildProcess._handle.onexit (internal/child_process.js:277:12)","name":"BoxError","reason":"External Error","details":{},"message":"undefined is not mounted"}}
Feb 23 09:40:14 box:tasks update 991: {"percent":100,"result":null,"error":{"stack":"BoxError: undefined is not mounted\n at /home/yellowtent/box/src/backuptask.js:709:29\n at f (/home/yellowtent/box/node_modules/once/once.js:25:25)\n at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:81:9)\n at ChildProcess.emit (events.js:400:28)\n at Process.ChildProcess._handle.onexit (internal/child_process.js:277:12)","name":"BoxError","reason":"External Error","details":{},"message":"undefined is not mounted"}}
undefined is not mounted
at /home/yellowtent/box/src/backuptask.js:709:29
at f (/home/yellowtent/box/node_modules/once/once.js:25:25)
at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:81:9)
at ChildProcess.emit (events.js:400:28)
at Process.ChildProcess._handle.onexit (internal/child_process.js:277:12)

screencapture-my-bebraver-cloud-2022-02-23-09_41_45.png

nebulon

@3246 the undefined is not mounted could be a bug. On which Cloudron version are you now?

I moved this to a new topic, as it seems like a separate issue.

BrutalBirdie

Hm I am running into the same issue. Cloudron v7.0.4

BrutalBirdie

2022-02-23T13:42:28.433Z box:locker Acquired : full_backup
2022-02-23T13:42:28.471Z box:tasks startTask - starting task 7545 with options {"timeout":86400000,"nice":15,"memoryLimit":800}. logs at /home/yellowtent/platformdata/logs/tasks/7545.log
2022-02-23T13:42:28.471Z box:shell startTask spawn: /usr/bin/sudo -S -E /home/yellowtent/box/src/scripts/starttask.sh 7545 /home/yellowtent/platformdata/logs/tasks/7545.log 15 800
2022-02-23T13:42:28.516Z box:shell startTask (stdout): Running as unit: box-task-7545.service

2022-02-23T13:42:29.340Z box:shell startTask (stdout): Finished with result: exit-code
Main processes terminated with: code=exited/status=50
Service runtime: 822ms

2022-02-23T13:42:29.341Z box:shell startTask code: 50, signal: null
2022-02-23T13:42:29.341Z box:tasks startTask: 7545 completed with code 50 and signal null
2022-02-23T13:42:29.342Z box:locker Released : full_backup
2022-02-23T13:42:29.342Z box:tasks startTask: 7545 done. error:  {
stack: 'BoxError: undefined is not mounted\n' +
'    at /home/yellowtent/box/src/backuptask.js:709:29\n' +
'    at f (/home/yellowtent/box/node_modules/once/once.js:25:25)\n' +
'    at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:81:9)\n' +
'    at ChildProcess.emit (events.js:400:28)\n' +
'    at Process.ChildProcess._handle.onexit (internal/child_process.js:277:12)',
name: 'BoxError',
reason: 'External Error',
details: {},
message: 'undefined is not mounted'
}

BrutalBirdie

Looking around in the box code:

grep -rin "is not mounted"

src/storage/filesystem.js:86:        if (result.mountpoint !== apiConfig.mountPoint) return callback(new BoxError(BoxError.FS_ERROR, `${apiConfig.mountPoint} is not mounted`));
src/storage/filesystem.js:88:        if (result.mountpoint === '/') return callback(new BoxError(BoxError.FS_ERROR, `${apiConfig.backupFolder} is not mounted`));
src/storage/filesystem.js:317:        if (!safe.child_process.execSync(`mountpoint -q -- ${apiConfig.mountPoint}`)) return callback(new BoxError(BoxError.BAD_FIELD, `${apiConfig.mountPoint} is not mounted`, { field: 'mountPoint' }));

The ${apiConfig.mountPoint} / ${apiConfig.backupFolder} => apiConfig is from api endpoint {{baseUrl}}/settings/backup_config I assume.

This api endpoint delivers:

{
    "provider": "cifs",
    "format": "tgz",
    "memoryLimit": 838860800,
    "schedulePattern": "00 00 5 * * *",
    "retentionPolicy": {
        "keepWithinSecs": 604800
    },
    "prefix": "",
    "noHardlinks": false,
    "mountOptions": {
        "host": "-REDACTED-.your-storagebox.de",
        "remoteDir": "/backup/cloudron-backup",
        "username": "-REDACTED-",
        "password": "●●●●●●●●"
    },
    "uploadPartSize": 10485760,
    "encryption": null,
    "mountStatus": {
        "state": "active",
        "message": "Mounted"
    }
}

There is no backupFolder here so either I am wrong with my guess about the api endpoint or something else.

Hmmm more digging.

BrutalBirdie

https://git.cloudron.io/cloudron/box/-/commit/a786fad3eed59e9ed141c7a3344466b61e280891#f873e27dcc2976565d4f0aca6c809f8d4e39c506_425_427

In src/settings.js deleted line 425:

backupConfig.mountPoint = '/mnt/cloudronbackup'; // hardcode the mountPoint for built-in mounts

This is a hint why I had to go with explicit /backup/cloudron-backup in https://forum.cloudron.io/topic/6529/shooting-a-cloudron-server-right-in-the-brain-i-deleted-appsdata-and-boxdata-by-accident because I am quite sure I never did this before.

nebulon

@BrutalBirdie so you are essentially saying the undefined comes from src/storage/filesystem.js:88 ?

Do you know what we have to setup and how to reproduce this issue?

For the /mnt/cloudronbackup, since there is only one backup backend supported, it made sense to hardcode this so it is less likely to clash with other directory names on the system. Did you hit any issue with this?

BrutalBirdie

The error is definitely in these 3 patches.

wget https://git.cloudron.io/cloudron/box/-/commit/04b7c14fd71e65072f43ed008d9e8b598dd946ab.diff
wget https://git.cloudron.io/cloudron/box/-/commit/8b9d821905b55c624084cb7665d8fbdb621a56e4.diff
wget https://git.cloudron.io/cloudron/box/-/commit/a786fad3eed59e9ed141c7a3344466b61e280891.diff

Because when I set the backup to no-op and then revert the changes:

git apply -R a786fad3eed59e9ed141c7a3344466b61e280891.diff
git apply -R 8b9d821905b55c624084cb7665d8fbdb621a56e4.diff
git apply -R 04b7c14fd71e65072f43ed008d9e8b598dd946ab.diff

And then setup my CIFS backup again, its working as intended.

Also regarding my previous post with explicit /backup/cloudron-backup in the now reverted version I only have to do /backup again

girish

Can you try updating to 7.1 and trying this? Not sure what the status of the "code" is with all these patches applied Would be great to know if this is reproducible with 7.1 . What you can do is to take a VM snapshot and then update without taking a cloudron backup.

BrutalBirdie

@girish this is my next testing step.

girish

@BrutalBirdie yes, please I did make sure to test all the variety of backends in 7.1, but let me know.

BrutalBirdie

Seems like after an upgrade to v7.1 there is no more issue.

So if you have to restore from v7.0.4 apply the 3 patches so you can restore Cloudron.
After the successful restore set the backup option to No-Op aka. No Backup!
De-apply the patches in reverse order.

git apply -R a786fad3eed59e9ed141c7a3344466b61e280891.diff
git apply -R 8b9d821905b55c624084cb7665d8fbdb621a56e4.diff
git apply -R 04b7c14fd71e65072f43ed008d9e8b598dd946ab.diff

Restart the box.service systemctl restart box.service
Then configure your backup again and make sure it works.

Then do the upgrade. Do not disable the automatic backup before the upgrade.

avatar1024

@BrutalBirdie Can you confirm which version you are on, 7.1.0 or 7.1.1? I've hit this issue after restoring an instance that was on 7.1.0 already.

BrutalBirdie

I am on v7.1.1 - now

This issue is very specific!
If you run into this and feel not comfortable fixing it alone, do not hesitate to write a support ticket or here in the forum!

The issue only appears after a specific error chain.

You need to have a Cloudron v7.0.4 which needs to be fully restored
Then you get the first error https://forum.cloudron.io/topic/6494/restore-error-cannot-access-error-before-initialization
2.1. solved by applying the 3x patches
Restored Cloudron running now and throwing the undefined is not mounted error when trying to backup
3.1. Still on version v7.0.4 you need to set backup option to No-Op No backup
3.2. De-apply the 3x patches in reverse order (like described above)
3.3. Restart box.service - systemctl restart box.service
3.4. Configure your backup solution again and make sure it works as intended
3.5. Upgrade to v7.1.1 and do not disable the pre-backup option
3.6. After the successful upgrade to v7.1.1 everything works as expected again

avatar1024

@BrutalBirdie Actually for me the issue appeared after:

Upgrading an instance to 7.1.0
Doing a full backup (which worked fine)
Restoring this instance on a new server (also worked fine) and shutting down the old server
But then daily backups on the newly restored instance fail with the "undefined is not mounted" error.

I've been in touch with support about this. I've given them ssh access and wait to hear what they say but perhaps simply upgrading to 7.1.1 might solve it.

girish

This was indeed a bug. Fixed in https://git.cloudron.io/cloudron/box/-/commit/2434e813836e7837162c1798794d04655227d9cf . Thanks a lot @avatar1024 . It will be part of 7.1.2

Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.

Solved backup error: "undefined is not mounted"