So, this is just a quick post because i'm still crying 
(but maybe it helps someone)
After (automatically) having my cloudron instance upgraded, i couldn't access the dashboard anymore, but (most of) the apps where still working. Also no CLI / remote access.
Edit and TL'DR:
Sorry, but it was very late.
TL'DR: the "addons" object in (every) cloudron app's cloudronManifest.json (especially custom, self made ones) is NOT OPTIONAL because of a bug in a migration file. You need to manually modify the apps in the mysql apps table. See below.
Okay, history:
The SSH troubleshoot command cloudron-support --troubleshoot revealed
root@xx /var/backups # cloudron-support --troubleshoot
Vendor: Gigabyte Technology Co., Ltd. Product: xx
Linux: xx
Ubuntu: noble 24.04
Processor: xx
BIOS Intel(R) Core(TM) i9xx
RAM: 131811572KB
Disk: /dev/md2 2.5T
[OK] node version is correct
[OK] IPv6 is enabled and public IPv6 address is working
[OK] docker is running
[OK] docker version is correct
[OK] MySQL is running
[OK] nginx is running
[OK] dashboard cert is valid
[FAIL] -> Could not load dashboard website with loopback check <-
root@xx /var/backups #
It was NOT a DNS or IP issue. (Cloudflare as DNS provider with no changes, all IP's/DNS's resolved correctly)
Checking the NGINX's logs revealed, the upstream connection failed to the /openid path:
Dec 16 16:10:16 xx nginx[146916]: 2025/12/16 16:10:16 [error] 146916#146916: *2528 connect() failed (111: Connection refused) while connecting to upstream, client: 172.18.16.72, server: my.xx.com, request: "GET /openid/.well-known[xx]>
Cool... (well not)...
The "dashboard" nginx configuration file for that location:
location ~ ^/openid/ {
proxy_pass http://127.0.0.1:3005;
}
okay, so port 3005... nothing running there (ss -ltnp | grep 3005)
hm, okay... the box.js should handle the start of the matching process, the logs revealed (crash loop):
(sudo tail -n 200 /home/yellowtent/platformdata/logs/box.log) :
2025-12-16T19:55:41.714Z box:server ==========================================
2025-12-16T19:55:41.715Z box:server Cloudron 9.0.13
2025-12-16T19:55:41.715Z box:server ==========================================
2025-12-16T19:55:41.715Z box:platform initialize: start platform
2025-12-16T19:55:41.716Z box:tasks stopAllTasks: 0 tasks are running. sending abort signal
2025-12-16T19:55:41.716Z box:shell tasks: /usr/bin/sudo --non-interactive /home/yellowtent/box/src/scripts/stoptask.sh all
2025-12-16T19:55:41.756Z box:locks releaseAll: all locks released
2025-12-16T19:55:41.758Z box:reverseproxy writeDashboardConfig: writing dashboard config for xxx.com
2025-12-16T19:55:41.763Z box:shell reverseproxy: /usr/bin/sudo --non-interactive /home/yellowtent/box/src/scripts/restartservice.sh nginx
2025-12-16T19:55:42.026Z box:updater notifyBoxUpdate: update finished from 8.3.2 to 9.0.13
2025-12-16T19:55:42.027Z Error starting servers
2025-12-16T19:55:42.027Z BoxError: Unknown column 'completed' in 'field list'
2025-12-16T19:55:42.027Z at Object.query (/home/yellowtent/box/src/database.js:96:22)
2025-12-16T19:55:42.027Z at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
2025-12-16T19:55:42.027Z at async list (/home/yellowtent/box/src/tasks.js:163:21)
2025-12-16T19:55:42.027Z at async Object.setCompletedByType (/home/yellowtent/box/src/tasks.js:172:21)
Okeee... Looks like a failed update... to be precise a failed migration...
"Unknown column "completed" in "field list" -> at tasks (lets check migrations related to tasks table) (/home/yellowtent/box/migrations)
voila -> 20250716130216-tasks-add-completed.js looks matching...
but that means, the migration did not run successfully...
lets check which ones we're missing:
cd /home/yellowtent/box
HOME=/home/yellowtent BOX_ENV=cloudron \
DATABASE_URL=mysql://root:password@127.0.0.1/box \
node_modules/.bin/db-migrate check
if your result IS NOT AN EMPTY ARRAY... (well, you have unapplied migrations).
Simplest: try to apply them. But lets kill the crash loop process first: sudo pkill -f /home/yellowtent/box/box.js
Okay:
cd /home/yellowtent/box
HOME=/home/yellowtent BOX_ENV=cloudron \
DATABASE_URL=mysql://root:password@127.0.0.1/box \
node_modules/.bin/db-migrate up
(well obviously they failed...).
mine was:
[ERROR] AssertionError [ERR_ASSERTION]: ifError got unwanted exception: Cannot read properties of undefined (reading 'oidc')
at /home/yellowtent/box/node_modules/db-migrate/lib/commands/on-complete.js:15:14
at tryCatcher (/home/yellowtent/box/node_modules/bluebird/js/release/util.js:16:23)
at Promise.successAdapter (/home/yellowtent/box/node_modules/bluebird/js/release/nodeify.js:22:30)
at Promise._settlePromise (/home/yellowtent/box/node_modules/bluebird/js/release/promise.js:601:21)
at Promise._settlePromiseCtx (/home/yellowtent/box/node_modules/bluebird/js/release/promise.js:641:10)
at _drainQueueStep (/home/yellowtent/box/node_modules/bluebird/js/release/async.js:97:12)
at _drainQueue (/home/yellowtent/box/node_modules/bluebird/js/release/async.js:86:9)
at Async._drainQueues (/home/yellowtent/box/node_modules/bluebird/js/release/async.js:102:5)
at Async.drainQueues [as _onImmediate] (/home/yellowtent/box/node_modules/bluebird/js/release/async.js:15:14)
at process.processImmediate (node:internal/timers:485:21)
at exports.up (/home/yellowtent/box/migrations/20250713133718-oidcClients-separate-ids-for-oidc-proxyauth.js:9:29)
-> Cannot read properties of undefined (reading 'oidc')
looking at the migration itself:
use strict';
exports.up = async function (db) {
const allApps = await db.runSql('SELECT * FROM apps');
for (const app of allApps) {
const manifest = JSON.parse(app.manifestJson);
// assume apps only have one of them
if (manifest.addons.oidc) {
await db.runSql('UPDATE oidcClients SET id=? WHERE id=?', [ ${app.id}-oidc, app.id ]);
} else {
await db.runSql('UPDATE oidcClients SET id=? WHERE id=?', [ ${app.id}-proxyauth, app.id ]);
}
}
};
exports.down = async function (db) {
};
AHA. They are trying to access the addons object in the cloudronManifest.json
Thats weird. Should be better escaped...
if (manifest.addons.oidc) { <- results in an error if addons is not defined.
(@cloudron: fix the migration with if (manifest.addons?.oidc))
Well... i obviously have apps installed which do not have addons object in the manifest, as not needed AND DECLARED AS OPTIONAL!!!
Lets check which ones (i have A LOT of custom apps..).
To check which ones we need to access the mysql db (as we're talking about migrations...), because we cannot remotely uninstall apps, as the CLI is not working (no auth process, which we found out) (By the way: the packaged cloudron-support on-side cli is really useless...)
lets connect to the mysql... how.. uff...
-> Ah, the start.sh script (/home/yellowtent/box/setup/start.sh) reveals an access, we can login with
sudo mysql -u root -ppassword
(yes, double "P")
maan, where are they... looking for the mysql cheatsheets.. (no guarantee for correct sql-syntax in this drunklatenightcryingbugfixsessionblogpost)
Okay. First, we know from the start.sh script the DB is called box, lets use it:
use box;
okay, we have an apps table (show tables;)... describing it reveals a manifestJson column, nice. (describe apps;)
So, lets get all apps and just check which manifestJson does not contain addons at all:
select id from apps where manifestJson is not null and manifestJson not like '%"addons"%';
okay... got 5 apps... well i did them quite some time ago... interestingly was one of them already un-installed ?! weird.
as i don't want to touch the migrations (would be a way easier fix, but not really longterm-stable, think backups, server-transfer etc), we just temporarily add an empty addons object to the manifest, so the migration-script "can try to find the oidc section. (whats it for anyway? need to research that...)
lets globally add the addons object if not present, do not want to do it row-by-row (which should usually be done in a live prod system with checks etc...)
(tired in the morning? try deleting a table in the prod DB, that will get you awake 
)
sql query to add an empty object for all rows where it doesnt exist(well, the word addons, but should be failsave, right?):
update apps set manifestJson = JSON_SET(manifestJson, '$.addons', JSON_OBJECT()) where manifestJson is not null and manifestJson not like '%"addons"%';
puh. okay. lets re-run the migrations:
cd /home/yellowtent/box
HOME=/home/yellowtent BOX_ENV=cloudron \
DATABASE_URL=mysql://root:password@127.0.0.1/box \
node_modules/.bin/db-migrate up
if you get no error ( i had another issue, but was my error (deleted a remote backupSite, don't ask....))
we verify that really all migrations are applied now:
node_modules/.bin/db-migrate check
should result in something like Migrations to run: []
NICE. Okay, let a final sudo reboot correctly start all systems again.
-> Done. (uih, the UI changed...)
(how to mark a topic/post as resolved here?! )
