Installation reverted to default - unable to restore using web interface

[phsc]

Hello,

For seemingly no reason, 1 of my Cloudron installations appears to have reverted to default last month. Looking at the logs it seems that I am currently running version V7.5.2.

When accessing apps, it says its not responding. When accessing the Cloudron setup (via my.domain.com), it comes up with the Domain Setup screen.

I attempted to use the restore link at the bottom of the Domain Setup screen (as per the restore clourdon instructions), but it just quickly takes me to a blank screen and then back to the Domain Setup screen.

I can see in the var/backups location I have 3 backups. Below is an extract from box.log since last restart, which mentions an error related to a bad field (no idea what that means). The local CLI is showing a cloud-int[1423] Used fallback datasource warning.

I had a look around with how to recover without the web interface, and it seems it might be possible with the Cloudron CLI, but I didn't find any instructions (in a post from 2017, it said there would be a future post on how to migrate a whole server, but I can't find it).

Any help you can provide would be appreciated.

Thanks,

Justin

box.log Extract

2023-10-02T23:01:14.576Z box:server ==========================================
2023-10-02T23:01:14.576Z box:server Cloudron 7.5.2
2023-10-02T23:01:14.576Z box:server ==========================================
2023-10-02T23:01:14.576Z box:platform initializing platform
2023-10-02T23:01:14.661Z box:tasks stopAllTasks: stopping all tasks
2023-10-02T23:01:14.661Z box:shell stopTask spawn: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/stoptask.sh all
2023-10-02T23:01:14.839Z box:shell stopTask (stdout): All tasks stopped

2023-10-02T23:01:14.905Z box:platform onActivated: starting post activation services
2023-10-02T23:01:14.906Z box:platform startInfra: checking infrastructure
2023-10-02T23:01:14.913Z box:platform startInfra: updating infrastructure from 49.4.0 to 49.5.0
2023-10-02T23:01:14.913Z box:locker Acquired : infra_start
2023-10-02T23:01:14.913Z box:platform removeAllContainers: removing all containers for infra upgrade
2023-10-02T23:01:14.913Z box:shell removeAllContainers exec: docker ps -qa --filter 'label=isCloudronManaged' | xargs --no-run-if-empty docker stop
2023-10-02T23:01:14.939Z box:shell removeAllContainers (stdout): null
2023-10-02T23:01:14.939Z box:shell removeAllContainers (stderr): null
2023-10-02T23:01:14.939Z box:shell removeAllContainers exec: docker ps -qa --filter 'label=isCloudronManaged' | xargs --no-run-if-empty docker rm -f
2023-10-02T23:01:14.956Z box:shell removeAllContainers (stdout): null
2023-10-02T23:01:14.956Z box:shell removeAllContainers (stderr): null
2023-10-02T23:01:14.956Z box:platform createDockerNetwork: recreating docker network
2023-10-02T23:01:14.956Z box:shell createDockerNetwork exec: docker network rm cloudron || true
2023-10-02T23:01:15.283Z box:shell createDockerNetwork (stdout): cloudron

2023-10-02T23:01:15.283Z box:shell createDockerNetwork (stderr): null
2023-10-02T23:01:15.283Z box:shell createDockerNetwork exec: docker network create --subnet=172.18.0.0/16 --ip-range=172.18.0.0/20 --gateway 172.18.0.1 --ipv6 --subnet=fd00:c107:d509::/64 cloudron
2023-10-02T23:01:15.387Z box:shell createDockerNetwork (stdout): 888a45f70a162a5aaaed4f2acefddf2b520684fdeff952d389ce9020616fe3c7

2023-10-02T23:01:15.387Z box:shell createDockerNetwork (stderr): null
2023-10-02T23:01:15.388Z box:platform markApps: reconfiguring apps
2023-10-02T23:01:15.388Z box:reverseproxy removeAppConfigs: removing app nginx configs
2023-10-02T23:01:15.409Z box:platform startInfra: Failed to start services. retry=false (attempt 0): ER_BAD_FIELD_ERROR: Unknown column 'apps.enableTurn' in 'field list'
2023-10-02T23:01:15.409Z box:platform BoxError: ER_BAD_FIELD_ERROR: Unknown column 'apps.enableTurn' in 'field list'
at Query.queryCallback (/home/yellowtent/box/src/database.js:91:38)
at Query.<anonymous> (/home/yellowtent/box/node_modules/mysql/lib/Connection.js:526:10)
at Query._callback (/home/yellowtent/box/node_modules/mysql/lib/Connection.js:488:16)
at Sequence.end (/home/yellowtent/box/node_modules/mysql/lib/protocol/sequences/Sequence.js:83:24)
at Query.ErrorPacket (/home/yellowtent/box/node_modules/mysql/lib/protocol/sequences/Query.js:92:8)
at Protocol._parsePacket (/home/yellowtent/box/node_modules/mysql/lib/protocol/Protocol.js:291:23)
at Parser._parsePacket (/home/yellowtent/box/node_modules/mysql/lib/protocol/Parser.js:433:10)
at Parser.write (/home/yellowtent/box/node_modules/mysql/lib/protocol/Parser.js:43:10)
at Protocol.write (/home/yellowtent/box/node_modules/mysql/lib/protocol/Protocol.js:38:16)
at Socket.<anonymous> (/home/yellowtent/box/node_modules/mysql/lib/Connection.js:88:28)
2023-10-02T23:06:14.937Z box:locker Lock unreleased infra_start
2023-10-02T23:11:14.959Z box:locker Lock unreleased infra_start
2023-10-02T23:16:14.986Z box:locker Lock unreleased infra_start
2023-10-02T23:21:14.996Z box:locker Lock unreleased infra_start
2023-10-02T23:26:15.004Z box:locker Lock unreleased infra_start

[girish]

@phsc this happens because the migrations did not complete for some reason. Please follow https://docs.cloudron.io/troubleshooting/#failed-upgrade

[phsc]

Hi @girish

Thanks for your response. Below is the steps I completed as per the guide you provided. Unfortunately I still have the domain setup page. Clicking the restore button still results in the same behavior as described earlier.

Domain: working
Nginx: running
start .sh: See extract below. Most notably "[ERROR] AssertionError" leading to "DB migration failed".
systemctl: running

Thanks for your help so far,

Justin

start .sh extract

administrator@phvculs:~$ sudo /home/yellowtent/box/setup/start.sh
[sudo] password for administrator:
2023-10-03T23:19:11 ==> start: Cloudron Start
media: x:500:
2023-10-03T23:19:12 ==> start: Configuring docker
Synchronizing state of apparmor.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable apparmor
2023-10-03T23:19:13 ==> start: Ensuring directories
2023-10-03T23:19:13 ==> start: Configuring journald
2023-10-03T23:19:13 ==> start: Setting up unbound
2023-10-03T23:19:15 ==> start: Adding systemd services
Synchronizing state of unbound.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable unbound
Synchronizing state of cron.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable cron
2023-10-03T23:19:19 ==> start: Configuring sudoers
2023-10-03T23:19:19 ==> start: Configuring collectd
2023-10-03T23:19:19 ==> start: Configuring sysctl
2023-10-03T23:19:19 ==> start: Configuring logrotate
2023-10-03T23:19:19 ==> start: Adding motd message for admins
2023-10-03T23:19:19 ==> start: Configuring nginx
2023-10-03T23:19:20 ==> start: Starting mysql
mysqladmin: [Warning] Using a password on the command line interface can be insecure.
Warning: Since password will be sent to server in plain text, use ssl connection to ensure password safety.
mysql: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.
2023-10-03T23:19:20 ==> start: Migrating data
Ignoring invalid configuration option passed to Connection: driver. This is currently a warning, but in future versions of MySQL2, an error will be thrown if you pass an invalid configuration option to a Connection
[ERROR] AssertionError [ERR_ASSERTION]: ifError got unwanted exception: Duplicate entry 'sftp_rsa_private_key' for key 'blobs.PRIMARY'
at /home/yellowtent/box/node_modules/db-migrate/lib/commands/on-complete.js:15:14
at tryCatcher (/home/yellowtent/box/node_modules/bluebird/js/release/util.js:16:23)
at Promise.successAdapter (/home/yellowtent/box/node_modules/bluebird/js/release/nodeify.js:22:30)
at Promise._settlePromise (/home/yellowtent/box/node_modules/bluebird/js/release/promise.js:601:21)
at Promise._settlePromiseCtx (/home/yellowtent/box/node_modules/bluebird/js/release/promise.js:641:10)
at _drainQueueStep (/home/yellowtent/box/node_modules/bluebird/js/release/async.js:97:12)
at _drainQueue (/home/yellowtent/box/node_modules/bluebird/js/release/async.js:86:9)
at Async._drainQueues (/home/yellowtent/box/node_modules/bluebird/js/release/async.js:102:5)
at Async.drainQueues [as _onImmediate] (/home/yellowtent/box/node_modules/bluebird/js/release/async.js:15:14)
at process.processImmediate (node:internal/timers:476:21)
at Packet.asError (/home/yellowtent/box/node_modules/mysql2/lib/packets/packet.js:728:17)
at Query.execute (/home/yellowtent/box/node_modules/mysql2/lib/commands/command.js:29:26)
at Connection.handlePacket (/home/yellowtent/box/node_modules/mysql2/lib/connection.js:456:32)
at PacketParser.onPacket (/home/yellowtent/box/node_modules/mysql2/lib/connection.js:85:12)
at PacketParser.executeStart (/home/yellowtent/box/node_modules/mysql2/lib/packet_parser.js:75:16)
at Socket.<anonymous> (/home/yellowtent/box/node_modules/mysql2/lib/connection.js:92:25)
at Socket.emit (node:events:513:28)
at addChunk (node:internal/streams/readable:324:12)
at readableAddChunk (node:internal/streams/readable:297:9)
at Readable.push (node:internal/streams/readable:234:10)
at TCP.onStreamRead (node:internal/stream_base_commons:190:23)
2023-10-03T23:19:21 ==> start: DB migration failed

[girish]

@phsc Please try the command at https://forum.cloudron.io/topic/10087/cloudron-shows-domain-setup-page-after-a-possible-crash/5 . You can also reach out to us on support@cloudron.io and we can do that for you. After you run the command, run the start.sh again and it should work.

[phsc]

Hi @girish
I ran the commands suggested and it has fixed the issue - thank you. Looking at the logs, the issue occurred when the update to version 7.5.2 was started. For the record, I did do a search in the forum for this issue, but didn't come across the article you referenced. Thanks again for your help.

[girish]

@phsc ha ha, no worries. I think the forum search can be far better as well .... I will make a pinned post for failed upgrades at some point.

[necrevistonnezr]

... or even in the docs (which are, by the way, excellent!)