Postgres not upgrading with 8.0.3
-
<30>1 2024-08-07T02:42:09Z srv.linuxsystems.io af04fd36-cd1b-4f63-851b-e079ebd10d34 804498 af04fd36-cd1b-4f63-851b-e079ebd10d34 - +442ms
tail: warning: --retry only effective for the initial open
2024-08-09T18:47:04.073Z box:shell getSwaps execArgs: swapon ["--noheadings","--raw","--bytes","--show=type,size,used,name"]
2024-08-09T18:47:06.747Z box:shell reboot /usr/bin/sudo -S /home/yellowtent/box/src/scripts/reboot.sh
2024-08-09T18:47:06.820Z box:box Received SIGTERM. Shutting down.
2024-08-09T18:47:06.822Z box:platform uninitializing platform
2024-08-09T18:47:06.824Z box:platform onDeactivated: stopping post activation services
2024-08-09T18:47:06.824Z box:tasks stopAllTasks: stopping all tasks
2024-08-09T18:47:06.824Z box:shell stopTask /usr/bin/sudo -S /home/yellowtent/box/src/scripts/stoptask.sh all
2024-08-09T18:47:06.866Z box:shell tail: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/logtail.sh --lines=100 --follow /home/yellowtent/platformdata/logs/af04fd36-cd1b-4f63-851b-e079ebd10d34/apptask.log /home/yellowtent/platformdata/logs/af04fd36-cd1b-4f63-851b-e079ebd10d34/app.log terminated
2024-08-09T18:47:06.867Z box:shell reboot: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/reboot.sh errored BoxError: reboot exited with code null signal SIGTERM
at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:122:19)
at ChildProcess.emit (node:events:518:28)
at ChildProcess.emit (node:domain:488:12)
at ChildProcess._handle.onexit (node:internal/child_process:294:12) {
reason: 'Shell Error',
details: {},
code: null,
signal: 'SIGTERM'
}
2024-08-09T18:47:06.867Z box:system reboot: could not reboot. BoxError: reboot exited with code null signal SIGTERM at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:122:19) at ChildProcess.emit (node:events:518:28) at ChildProcess.emit (node:domain:488:12) at ChildProcess.handle.onexit (node:internal/child_process:294:12) { reason: 'Shell Error', details: {}, code: null, signal: 'SIGTERM' }
All tasks stopped
2024-08-09T18:51:00.471Z box:server ==========================================
2024-08-09T18:51:00.471Z box:server Cloudron 8.0.3
2024-08-09T18:51:00.471Z box:server ==========================================
2024-08-09T18:51:00.471Z box:platform initialize: start platform
2024-08-09T18:51:01.411Z box:tasks stopAllTasks: stopping all tasks
2024-08-09T18:51:01.412Z box:shell stopTask /usr/bin/sudo -S /home/yellowtent/box/src/scripts/stoptask.sh all
All tasks stopped
2024-08-09T18:51:02.002Z box:reverseproxy writeDashboardConfig: writing dashboard config for linuxsystems.io
2024-08-09T18:51:02.258Z box:shell isOscpEnabled execArgs: openssl ["x509","-in","/home/yellowtent/platformdata/nginx/cert/.linuxsystems.io.cert","-noout","-ocsp_uri"]
2024-08-09T18:51:02.301Z box:shell reload /usr/bin/sudo -S /home/yellowtent/box/src/scripts/restartservice.sh nginx
2024-08-09T18:51:04.993Z box:oidc Using existing OIDC EdDSA key
2024-08-09T18:51:04.994Z box:oidc Using existing OIDC RS256 key
2024-08-09T18:51:04.999Z box:oidc start: create provider for my.linuxsystems.io at /openid
2024-08-09T18:51:05.236Z box:platform onActivated: starting post activation services
2024-08-09T18:51:05.236Z box:platform startInfra: checking infrastructure
2024-08-09T18:51:05.285Z box:platform startInfra: updating infrastructure from 49.6.0 to 49.7.0
2024-08-09T18:51:05.285Z box:locker Acquired : infra_start
2024-08-09T18:51:05.285Z box:platform removeAllContainers: removing all containers for infra upgrade
2024-08-09T18:51:05.285Z box:shell removeAllContainers exec: docker ps -qa --filter 'label=isCloudronManaged' | xargs --no-run-if-empty docker stop
2024-08-09T18:51:40.952Z box:shell removeAllContainers exec: docker ps -qa --filter 'label=isCloudronManaged' | xargs --no-run-if-empty docker rm -f
2024-08-09T18:51:41.733Z box:platform createDockerNetwork: recreating docker network
2024-08-09T18:51:41.733Z box:shell createDockerNetwork execArgs: docker ["network","rm","-f","cloudron"]
2024-08-09T18:51:41.754Z box:shell createDockerNetwork: docker with args network rm -f cloudron errored Error: Command failed: docker network rm -f cloudron
Error response from daemon: error while removing network: network cloudron id f11bc93f95b2d61a8856e86957473fd57d04ffe57cae32b177d32207459a1580 has active endpointsat genericNodeError (node:internal/errors:984:15) at wrappedFn (node:internal/errors:538:14) at ChildProcess.exithandler (node:child_process:422:12) at ChildProcess.emit (node:events:518:28) at maybeClose (node:internal/child_process:1105:16) at ChildProcess._handle.onexit (node:internal/child_process:305:5) {
code: 1,
killed: false,
signal: null,
cmd: 'docker network rm -f cloudron'
}
2024-08-09T18:51:41.758Z box:platform startInfra: Failed to start services. retry=false (attempt 0): createDockerNetwork errored with code 1 message Command failed: docker network rm -f cloudron
Error response from daemon: error while removing network: network cloudron id f11bc93f95b2d61a8856e86957473fd57d04ffe57cae32b177d32207459a1580 has active endpoints2024-08-09T18:51:41.759Z box:platform BoxError: createDockerNetwork errored with code 1 message Command failed: docker network rm -f cloudron
Error response from daemon: error while removing network: network cloudron id f11bc93f95b2d61a8856e86957473fd57d04ffe57cae32b177d32207459a1580 has active endpointsat /home/yellowtent/box/src/shell.js:38:23 at ChildProcess.exithandler (node:child_process:430:5) at ChildProcess.emit (node:events:518:28) at maybeClose (node:internal/child_process:1105:16) at ChildProcess._handle.onexit (node:internal/child_process:305:5)
2024-08-09T18:52:23.863Z box:shell getSwaps execArgs: swapon ["--noheadings","--raw","--bytes","--show=type,size,used,name"]
HttpError: Unknown error: 400 <html>
<head><title>400 Bad Request</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
<hr><center>nginx</center>
</body>
</html>at getSystemGraphs (/home/yellowtent/box/src/routes/system.js:126:28) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
status: 500,
internalError: BoxError: Unknown error: 400 <html>
<head><title>400 Bad Request</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
<hr><center>nginx</center>
</body>
</html>at Object.getSystem (/home/yellowtent/box/src/graphs.js:106:46) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) { reason: 'External Error', details: {}
},
details: {}
}
2024-08-09T18:52:57.368Z box:shell getSwaps execArgs: swapon ["--noheadings","--raw","--bytes","--show=type,size,used,name"]
2024-08-09T18:52:57.508Z box:shell hasAVX execArgs: grep ["-q","avx","/proc/cpuinfo"]
2024-08-09T18:52:57.569Z box:shell statusNginx execArgs: systemctl ["is-active","nginx"]
2024-08-09T18:52:57.572Z box:shell statusUnbound execArgs: systemctl ["is-active","unbound"]
2024-08-09T18:53:11.953Z box:services getServiceLogs: getting logs for postgresql
2024-08-09T18:56:05.285Z box:locker Lock unreleased infra_start
2024-08-09T19:01:05.286Z box:locker Lock unreleased infra_start
2024-08-09T19:04:12.215Z box:shell hasAVX execArgs: grep ["-q","avx","/proc/cpuinfo"]
2024-08-09T19:04:12.278Z box:shell statusNginx execArgs: systemctl ["is-active","nginx"]
2024-08-09T19:04:12.281Z box:shell statusUnbound execArgs: systemctl ["is-active","unbound"]
2024-08-09T19:05:13.665Z box:shell hasAVX execArgs: grep ["-q","avx","/proc/cpuinfo"]
2024-08-09T19:05:13.730Z box:shell statusUnbound execArgs: systemctl ["is-active","unbound"]
2024-08-09T19:05:13.734Z box:shell statusNginx execArgs: systemctl ["is-active","nginx"]
2024-08-09T19:06:05.287Z box:locker Lock unreleased infra_start -
I've got sort of the same thing here, but the ip(v6) address the system appears to be using doesn't look like the one my dedicated server should be using based on my account info.
---my info---
- IPv6 Address 2605:9f80:1000:505::2/64
- IPv6 Gateway 2605:9f80:1000:505::1
console:
root@srv:~# ip -f inet6 -br addr
lo UNKNOWN ::1/128
primary UP fe80::225:90ff:fea3:eb00/64
br-f11bc93f95b2 UP fd00:c107:d509::1/64 fe80::42:19ff:fe21:3d66/64 fe80::1/64
veth03fbb82@if33 UP fe80::6ca7:1bff:febd:c2f3/64
vethe35151d@if35 UP fe80::9c23:4fff:fe26:fb64/64
I have reached out to the datacenter however I don't think anything has changed except the upgrade to 8.0.3 which made apps go unresponsible on trying to edit/manage them and then after a reboot, nothing is coming up at all.
Support SSH is enabled though in case anyone wants to have a look...
-
Looked like my IPv6 setup had never been used and it was a NR address on the primary interface. I've added my info and most services will start when I reload them on the services page, so far postgres has not been able to start though:
Aug 09 11:44:05 ALTER ROLE
Aug 09 11:44:05 ==> Upgrading pgvectors
Aug 09 11:44:05 ERROR: database "cloudronpgvectorupdate" already exists
Aug 09 11:44:05 2024-08-09 18:44:05.144 UTC [31] root@postgres ERROR: database "cloudronpgvectorupdate" already exists
Aug 09 11:44:05 2024-08-09 18:44:05.144 UTC [31] root@postgres STATEMENT: CREATE DATABASE cloudronpgvectorupdate
Aug 09 11:44:12 ==> Detected existing installation
Aug 09 11:44:12 ==> Copying updated config
Aug 09 11:44:12 ==> Updating existing postgresql
Aug 09 11:44:12 waiting for server to start....2024-08-09 18:44:12.499 GMT [15] LOG: skipping missing configuration file "/var/lib/postgresql/14/main/custom.conf"
Aug 09 11:44:12 2024-08-09 18:44:12.499 GMT [15] LOG: skipping missing configuration file "/var/lib/postgresql/14/main/custom.conf"
Aug 09 11:44:12 2024-08-09 18:44:12.591 UTC [15] LOG: starting PostgreSQL 14.12 (Ubuntu 14.12-0ubuntu0.22.04.1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0, 64-bit
Aug 09 11:44:12 2024-08-09 18:44:12.591 UTC [15] LOG: listening on IPv4 address "0.0.0.0", port 5432
Aug 09 11:44:12 2024-08-09 18:44:12.591 UTC [15] LOG: listening on IPv6 address "::", port 5432
Aug 09 11:44:12 2024-08-09 18:44:12.619 UTC [15] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
Aug 09 11:44:12 2024-08-09 18:44:12.655 UTC [16] LOG: database system was interrupted; last known up at 2024-08-09 18:44:04 UTC
Aug 09 11:44:13 .2024-08-09 18:44:13.563 UTC [16] LOG: database system was not properly shut down; automatic recovery in progress
Aug 09 11:44:13 2024-08-09 18:44:13.599 UTC [16] LOG: redo starts at 5/92EC52D0
Aug 09 11:44:13 2024-08-09 18:44:13.599 UTC [16] LOG: invalid record length at 5/92EC6930: wanted 24, got 0
Aug 09 11:44:13 2024-08-09 18:44:13.599 UTC [16] LOG: redo done at 5/92EC68D8 system usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s
Aug 09 11:44:13 2024-08-09 18:44:13.918 UTC [15] LOG: database system is ready to accept connections
Aug 09 11:44:13 done
Aug 09 11:44:13 server started
Aug 09 11:44:13 ==> Waiting for Postgres
Aug 09 11:44:14 pg_ctl: server is running (PID: 15)
Aug 09 11:44:14 /usr/lib/postgresql/14/bin/postgres "-D" "/var/lib/postgresql/14/main"
Aug 09 11:44:14 ==> Postgres is running
Aug 09 11:44:14 ==> Resetting root password
Aug 09 11:44:14 ALTER ROLE
Aug 09 11:44:14 ==> Upgrading pgvectors
Aug 09 11:44:14 ERROR: database "cloudronpgvectorupdate" already exists
Aug 09 11:44:14 2024-08-09 18:44:14.131 UTC [31] root@postgres ERROR: database "cloudronpgvectorupdate" already exists
Aug 09 11:44:14 2024-08-09 18:44:14.131 UTC [31] root@postgres STATEMENT: CREATE DATABASE cloudronpgvectorupdate
Aug 09 11:44:27 ==> Detected existing installation
Aug 09 11:44:27 ==> Copying updated config
Aug 09 11:44:27 ==> Updating existing postgresql
Aug 09 11:44:27 waiting for server to start....2024-08-09 18:44:27.737 GMT [14] LOG: skipping missing configuration file "/var/lib/postgresql/14/main/custom.conf"
Aug 09 11:44:27 2024-08-09 18:44:27.737 GMT [14] LOG: skipping missing configuration file "/var/lib/postgresql/14/main/custom.conf"
Aug 09 11:44:27 2024-08-09 18:44:27.832 UTC [14] LOG: starting PostgreSQL 14.12 (Ubuntu 14.12-0ubuntu0.22.04.1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0, 64-bit
Aug 09 11:44:27 2024-08-09 18:44:27.833 UTC [14] LOG: listening on IPv4 address "0.0.0.0", port 5432
Aug 09 11:44:27 2024-08-09 18:44:27.833 UTC [14] LOG: listening on IPv6 address "::", port 5432
Aug 09 11:44:27 2024-08-09 18:44:27.898 UTC [14] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
Aug 09 11:44:27 2024-08-09 18:44:27.968 UTC [15] LOG: database system was interrupted; last known up at 2024-08-09 18:44:13 UTC
Aug 09 11:44:28 .2024-08-09 18:44:28.780 UTC [15] LOG: database system was not properly shut down; automatic recovery in progress
Aug 09 11:44:28 2024-08-09 18:44:28.802 UTC [15] LOG: redo starts at 5/92EC69A8
Aug 09 11:44:28 2024-08-09 18:44:28.802 UTC [15] LOG: invalid record length at 5/92EC8108: wanted 24, got 0
Aug 09 11:44:28 2024-08-09 18:44:28.802 UTC [15] LOG: redo done at 5/92EC80B0 system usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s
Aug 09 11:44:29 2024-08-09 18:44:29.132 UTC [14] LOG: database system is ready to accept connections
Aug 09 11:44:29 done
Aug 09 11:44:29 server started
Aug 09 11:44:29 ==> Waiting for Postgres
Aug 09 11:44:29 pg_ctl: server is running (PID: 14)
Aug 09 11:44:29 /usr/lib/postgresql/14/bin/postgres "-D" "/var/lib/postgresql/14/main"
Aug 09 11:44:29 ==> Postgres is running
Aug 09 11:44:29 ==> Resetting root password
Aug 09 11:44:29 ALTER ROLE
Aug 09 11:44:29 ==> Upgrading pgvectors
Aug 09 11:44:29 2024-08-09 18:44:29.370 UTC [30] root@postgres ERROR: database "cloudronpgvectorupdate" already exists
Aug 09 11:44:29 ERROR: database "cloudronpgvectorupdate" already exists
Aug 09 11:44:29 2024-08-09 18:44:29.370 UTC [30] root@postgres STATEMENT: CREATE DATABASE cloudronpgvectorupdate -
...and that was going nowhere, a reboot leaves services in the red and still:
Aug 09 11:45:51 ERROR: database "cloudronpgvectorupdate" already exists
Aug 09 11:45:51 2024-08-09 18:45:51.403 UTC [31] root@postgres STATEMENT: CREATE DATABASE cloudronpgvectorupdate
Aug 09 11:46:52 ==> Detected existing installation
Aug 09 11:46:52 ==> Copying updated config
Aug 09 11:46:52 ==> Updating existing postgresql
Aug 09 11:46:52 waiting for server to start....2024-08-09 18:46:52.276 GMT [15] LOG: skipping missing configuration file "/var/lib/postgresql/14/main/custom.conf"
Aug 09 11:46:52 2024-08-09 18:46:52.276 GMT [15] LOG: skipping missing configuration file "/var/lib/postgresql/14/main/custom.conf"
Aug 09 11:46:52 2024-08-09 18:46:52.384 UTC [15] LOG: starting PostgreSQL 14.12 (Ubuntu 14.12-0ubuntu0.22.04.1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0, 64-bit
Aug 09 11:46:52 2024-08-09 18:46:52.384 UTC [15] LOG: listening on IPv4 address "0.0.0.0", port 5432
Aug 09 11:46:52 2024-08-09 18:46:52.384 UTC [15] LOG: listening on IPv6 address "::", port 5432
Aug 09 11:46:52 2024-08-09 18:46:52.456 UTC [15] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
Aug 09 11:46:52 2024-08-09 18:46:52.516 UTC [16] LOG: database system was interrupted; last known up at 2024-08-09 18:45:51 UTC
Aug 09 11:46:53 .2024-08-09 18:46:53.643 UTC [16] LOG: database system was not properly shut down; automatic recovery in progress
Aug 09 11:46:53 2024-08-09 18:46:53.662 UTC [16] LOG: redo starts at 5/92ECB418
Aug 09 11:46:53 2024-08-09 18:46:53.662 UTC [16] LOG: invalid record length at 5/92ECCE78: wanted 24, got 0
Aug 09 11:46:53 2024-08-09 18:46:53.662 UTC [16] LOG: redo done at 5/92ECCE20 system usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s
Aug 09 11:46:53 2024-08-09 18:46:53.922 UTC [15] LOG: database system is ready to accept connections
Aug 09 11:46:53 done
Aug 09 11:46:53 server started
Aug 09 11:46:53 ==> Waiting for Postgres
Aug 09 11:46:53 pg_ctl: server is running (PID: 15)
Aug 09 11:46:53 /usr/lib/postgresql/14/bin/postgres "-D" "/var/lib/postgresql/14/main"
Aug 09 11:46:53 ==> Postgres is running
Aug 09 11:46:53 ==> Resetting root password
Aug 09 11:46:54 ALTER ROLE
Aug 09 11:46:54 ==> Upgrading pgvectors
Aug 09 11:46:54 2024-08-09 18:46:54.092 UTC [31] root@postgres ERROR: database "cloudronpgvectorupdate" already exists
Aug 09 11:46:54 2024-08-09 18:46:54.092 UTC [31] root@postgres STATEMENT: CREATE DATABASE cloudronpgvectorupdate
Aug 09 11:46:54 ERROR: database "cloudronpgvectorupdate" already exists
Aug 09 11:51:17 ==> Detected existing installation
Aug 09 11:51:18 ==> Copying updated config
Aug 09 11:51:18 ==> Updating existing postgresql
Aug 09 11:51:22 waiting for server to start....2024-08-09 18:51:22.965 GMT [14] LOG: skipping missing configuration file "/var/lib/postgresql/14/main/custom.conf"
Aug 09 11:51:22 2024-08-09 18:51:22.965 GMT [14] LOG: skipping missing configuration file "/var/lib/postgresql/14/main/custom.conf"
Aug 09 11:51:25 ..2024-08-09 18:51:25.722 UTC [14] LOG: starting PostgreSQL 14.12 (Ubuntu 14.12-0ubuntu0.22.04.1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0, 64-bit
Aug 09 11:51:25 2024-08-09 18:51:25.722 UTC [14] LOG: listening on IPv4 address "0.0.0.0", port 5432
Aug 09 11:51:25 2024-08-09 18:51:25.722 UTC [14] LOG: listening on IPv6 address "::", port 5432
Aug 09 11:51:26 ..2024-08-09 18:51:26.995 UTC [14] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
Aug 09 11:51:27 2024-08-09 18:51:27.683 UTC [15] LOG: database system was interrupted; last known up at 2024-08-09 18:46:53 UTC
Aug 09 11:51:32 ..... -
@cygnostik said in Postgres not upgrading with 8.0.3:
Support SSH is enabled though in case anyone wants to have a look...
Can take a look but you have to send us a mail to support@cloudron.io . Can you quickly drop a mail with your Domain name/IP address?
-
-