mysql error: ER_BAD_FIELD_ERROR: Unknown column 'apps.label' in 'field list'

root_iam

hi,

since yesterday I cannot access the cloudron dashboard anymore. "Error. Something has gone wrong".

Tha box service and the mysql service are running fine.

In the log /home/yellowtent/platformdata/logs/box.log I can see the following errors:

2019-05-16T19:09:15.900Z ==========================================
2019-05-16T19:09:15.900Z  Cloudron will use the following settings
2019-05-16T19:09:15.900Z ==========================================
2019-05-16T19:09:15.900Z
2019-05-16T19:09:15.900Z  Environment:                     CLOUDRON
2019-05-16T19:09:15.900Z  Version:                         4.0.3
2019-05-16T19:09:15.900Z  Admin Origin:                    https://my.xxx.xx
2019-05-16T19:09:15.900Z  Appstore API server origin:      https://api.cloudron.io
2019-05-16T19:09:15.901Z  Appstore Web server origin:      https://cloudron.io
2019-05-16T19:09:15.901Z  SysAdmin Port:                   3001
2019-05-16T19:09:15.901Z  LDAP Server Port:                3002
2019-05-16T19:09:15.901Z  Docker Proxy Port:               3003
2019-05-16T19:09:15.901Z
2019-05-16T19:09:15.901Z ==========================================
2019-05-16T19:09:15.901Z
2019-05-16T19:09:15.910Z Warning: connect.session() MemoryStore is not
designed for a production environment, as it will leak
memory, and will not scale past a single process.
2019-05-16T19:09:16.087Z box:shell reload spawn: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/reloadnginx.sh
2019-05-16T19:09:16.149Z box:platform initializing addon infrastructure
2019-05-16T19:09:16.155Z box:platform Updating infrastructure from 48.12.1 to 48.14.0
2019-05-16T19:09:16.155Z box:locker Acquired : platform_start
2019-05-16T19:09:16.155Z box:platform stopping all containers for infra upgrade
2019-05-16T19:09:16.155Z box:shell stopContainers exec: docker ps -qa --filter 'network=cloudron' | xargs --no-run-if-empty docker stop
2019-05-16T19:09:16.189Z box:dockerproxy startDockerProxy: started proxy on port 3003
2019-05-16T19:09:16.190Z Cloudron is up and running
2019-05-16T19:09:16.199Z box:shell reload spawn: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/reloadnginx.sh
2019-05-16T19:09:16.203Z box:reverseproxy configureDefaultServer: done
2019-05-16T19:09:21.581Z box:shell stopContainers (stdout):
2019-05-16T19:09:21.582Z box:shell stopContainers (stderr):
2019-05-16T19:09:21.582Z box:shell stopContainers exec: docker ps -qa --filter 'network=cloudron' | xargs --no-run-if-empty docker rm -f
2019-05-16T19:09:21.653Z box:shell stopContainers (stdout):
2019-05-16T19:09:21.653Z box:shell stopContainers (stderr):
2019-05-16T19:09:21.653Z box:platform startApps: reconfiguring installed apps
2019-05-16T19:09:21.765Z box:cloudron AppsError: Internal error
    at /home/yellowtent/box/src/apps.js:1413:36
    at /home/yellowtent/box/src/apps.js:521:31
    at /home/yellowtent/box/src/appdb.js:243:27
    at Query.args.(anonymous function) (/home/yellowtent/box/src/database.js:154:9)
    at Query.<anonymous> (/home/yellowtent/box/node_modules/mysql/lib/Connection.js:525:10)
    at Query._callback (/home/yellowtent/box/node_modules/mysql/lib/Connection.js:491:16)
    at Query.Sequence.end (/home/yellowtent/box/node_modules/mysql/lib/protocol/sequences/Sequence.js:83:24)
    at Query.ErrorPacket (/home/yellowtent/box/node_modules/mysql/lib/protocol/sequences/Query.js:90:8)
    at Protocol._parsePacket (/home/yellowtent/box/node_modules/mysql/lib/protocol/Protocol.js:291:23)
    at Parser._parsePacket (/home/yellowtent/box/node_modules/mysql/lib/protocol/Parser.js:433:10)
2019-05-16T17:10:00.069Z ERROR Internal error [ /home/yellowtent/box/node_modules/connect-lastmile/lib/index.js:37:25 ]
stack:
  """
    HttpError: Internal error
        at /home/yellowtent/box/src/routes/apps.js:59:32
        at /home/yellowtent/box/src/apps.js:535:27
        at /home/yellowtent/box/src/apps.js:521:31
        at /home/yellowtent/box/src/appdb.js:243:27
        at Query.args.(anonymous function) (/home/yellowtent/box/src/database.js:154:9)
        at Query.<anonymous> (/home/yellowtent/box/node_modules/mysql/lib/Connection.js:525:10)
        at Query._callback (/home/yellowtent/box/node_modules/mysql/lib/Connection.js:491:16)
        at Query.Sequence.end (/home/yellowtent/box/node_modules/mysql/lib/protocol/sequences/Sequence.js:83:24)
        at Query.ErrorPacket (/home/yellowtent/box/node_modules/mysql/lib/protocol/sequences/Query.js:90:8)
        at Protocol._parsePacket (/home/yellowtent/box/node_modules/mysql/lib/protocol/Protocol.js:291:23)
  """
name:          HttpError
status:        500
message:       Internal error
internalError:
  """
    stack:
      """
        AppsError: Internal error
            at /home/yellowtent/box/src/apps.js:521:40
            at /home/yellowtent/box/src/appdb.js:243:27
            at Query.args.(anonymous function) (/home/yellowtent/box/src/database.js:154:9)
            at Query.<anonymous> (/home/yellowtent/box/node_modules/mysql/lib/Connection.js:525:10)
            at Query._callback (/home/yellowtent/box/node_modules/mysql/lib/Connection.js:491:16)
            at Query.Sequence.end (/home/yellowtent/box/node_modules/mysql/lib/protocol/sequences/Sequence.js:83:24)
            at Query.ErrorPacket (/home/yellowtent/box/node_modules/mysql/lib/protocol/sequences/Query.js:90:8)
            at Protocol._parsePacket (/home/yellowtent/box/node_modules/mysql/lib/protocol/Protocol.js:291:23)
            at Parser._parsePacket (/home/yellowtent/box/node_modules/mysql/lib/protocol/Parser.js:433:10)
            at Parser.write (/home/yellowtent/box/node_modules/mysql/lib/protocol/Parser.js:43:10)
      """
    name:        AppsError
    reason:      Internal Error
    message:     Internal error
    nestedError:
      """
        stack:
          """
            DatabaseError: Internal error
                at /home/yellowtent/box/src/appdb.js:243:36
                at Query.args.(anonymous function) (/home/yellowtent/box/src/database.js:154:9)
                at Query.<anonymous> (/home/yellowtent/box/node_modules/mysql/lib/Connection.js:525:10)
                at Query._callback (/home/yellowtent/box/node_modules/mysql/lib/Connection.js:491:16)
                at Query.Sequence.end (/home/yellowtent/box/node_modules/mysql/lib/protocol/sequences/Sequence.js:83:24)
                at Query.ErrorPacket (/home/yellowtent/box/node_modules/mysql/lib/protocol/sequences/Query.js:90:8)
                at Protocol._parsePacket (/home/yellowtent/box/node_modules/mysql/lib/protocol/Protocol.js:291:23)
                at Parser._parsePacket (/home/yellowtent/box/node_modules/mysql/lib/protocol/Parser.js:433:10)
                at Parser.write (/home/yellowtent/box/node_modules/mysql/lib/protocol/Parser.js:43:10)
                at Protocol.write (/home/yellowtent/box/node_modules/mysql/lib/protocol/Protocol.js:38:16)
          """
        name:        DatabaseError
        reason:      Internal error
        message:     Internal error
        nestedError:
          """
            stack:
              """
                Error: ER_BAD_FIELD_ERROR: Unknown column 'apps.label' in 'field list'
                    at Query.Sequence._packetToError (/home/yellowtent/box/node_modules/mysql/lib/protocol/sequences/Sequence.js:47:14)
                    at Query.ErrorPacket (/home/yellowtent/box/node_modules/mysql/lib/protocol/sequences/Query.js:77:18)
                    at Protocol._parsePacket (/home/yellowtent/box/node_modules/mysql/lib/protocol/Protocol.js:291:23)
                    at Parser._parsePacket (/home/yellowtent/box/node_modules/mysql/lib/protocol/Parser.js:433:10)
                    at Parser.write (/home/yellowtent/box/node_modules/mysql/lib/protocol/Parser.js:43:10)
                    at Protocol.write (/home/yellowtent/box/node_modules/mysql/lib/protocol/Protocol.js:38:16)
                    at Socket.<anonymous> (/home/yellowtent/box/node_modules/mysql/lib/Connection.js:91:28)
                    at Socket.<anonymous> (/home/yellowtent/box/node_modules/mysql/lib/Connection.js:525:10)
                    at Socket.emit (events.js:189:13)
                    at addChunk (_stream_readable.js:284:12)
                    --------------------
                    at Protocol._enqueue (/home/yellowtent/box/node_modules/mysql/lib/protocol/Protocol.js:144:48)
                    at PoolConnection.query (/home/yellowtent/box/node_modules/mysql/lib/Connection.js:201:25)
                    at Object.query (/home/yellowtent/box/src/database.js:157:30)
                    at Object.getAll (/home/yellowtent/box/src/appdb.js:235:14)
                    at /home/yellowtent/box/src/apps.js:520:15
                    at /home/yellowtent/box/src/apps.js:449:9
                    at /home/yellowtent/box/src/domaindb.js:52:9
                    at Query.args.(anonymous function) (/home/yellowtent/box/src/database.js:154:9)
                    at Query.<anonymous> (/home/yellowtent/box/node_modules/mysql/lib/Connection.js:525:10)
                    at Query._callback (/home/yellowtent/box/node_modules/mysql/lib/Connection.js:491:16)
              """
            message:    ER_BAD_FIELD_ERROR: Unknown column 'apps.label' in 'field list'
            code:       ER_BAD_FIELD_ERROR
            errno:      1054
            sqlMessage: Unknown column 'apps.label' in 'field list'
            sqlState:   42S22
            index:      0
            sql:        SELECT apps.id,apps.appStoreId,apps.installationState,apps.installationProgress,apps.runState,apps.health,apps.containerId,apps.manifestJson,apps.httpPort,subdomains.subdomain AS location,subdomains.domain,apps.accessRestrictionJson,apps.restoreConfigJson,apps
.oldConfigJson,apps.updateConfigJson,apps.memoryLimit,apps.label,apps.tagsJson,apps.xFrameOptions,apps.sso,apps.debugModeJson,apps.robotsTxt,apps.enableBackup,apps.creationTime,apps.updateTime,apps.ownerId,apps.mailboxName,apps.enableAutomaticUpdate,apps.dataDir,apps.ts,apps.healthTi
me,GROUP_CONCAT(CAST(appPortBindings.hostPort AS CHAR(6))) AS hostPorts, GROUP_CONCAT(appPortBindings.environmentVariable) AS environmentVariables, GROUP_CONCAT(appPortBindings.type) AS portTypes,JSON_ARRAYAGG(appEnvVars.name) AS envNames, JSON_ARRAYAGG(appEnvVars.value) AS envValues
 FROM apps  LEFT OUTER JOIN appPortBindings ON apps.id = appPortBindings.appId  LEFT OUTER JOIN appEnvVars ON apps.id = appEnvVars.appId  LEFT OUTER JOIN subdomains ON apps.id = subdomains.appId AND subdomains.type = 'primary' GROUP BY apps.id ORDER BY apps.id
          """
      """
  """
Box GET /api/v1/apps 500 5.843 ms - 70

I dont't know what to about it. Could somebody help me?

Thanks a lot!

murgero

A silly response maybe, but have you tried rebooting it?

root_iam

Yes, tried that a few times. Also restarting the services didn;t help.

nebulon

Apparently the database migration didn't succeed during the latest update for some reason. We have seen this in the past when the server is running out of main memory.
To fix this issue, SSH into the server and run /home/yellowtent/box/setup/start.sh as root. This should ensure the database schema gets updated, ensures proper permissions and it also restarts all relevant services.

root_iam

Thanks for your reply.

I tried running the script:

==> Cloudron Start
==> Configuring docker
apparmor.service is not a native service, redirecting to systemd-sysv-install
Executing /lib/systemd/systemd-sysv-install enable apparmor
Error response from daemon: network with name cloudron already exists
==> Ensuring directories
==> Configuring journald
==> Setting up unbound

The error is still there. The logs are showing the same entries as before:

2019-05-17T07:52:08.904Z ==========================================
2019-05-17T07:52:08.904Z  Cloudron will use the following settings
2019-05-17T07:52:08.904Z ==========================================
2019-05-17T07:52:08.904Z
2019-05-17T07:52:08.904Z  Environment:                     CLOUDRON
2019-05-17T07:52:08.904Z  Version:                         4.0.3
2019-05-17T07:52:08.904Z  Admin Origin:                    https://my.xxx.xx
2019-05-17T07:52:08.904Z  Appstore API server origin:      https://api.cloudron.io
2019-05-17T07:52:08.904Z  Appstore Web server origin:      https://cloudron.io
2019-05-17T07:52:08.904Z  SysAdmin Port:                   3001
2019-05-17T07:52:08.905Z  LDAP Server Port:                3002
2019-05-17T07:52:08.905Z  Docker Proxy Port:               3003
2019-05-17T07:52:08.905Z
2019-05-17T07:52:08.905Z ==========================================
2019-05-17T07:52:08.905Z
2019-05-17T07:52:08.912Z Warning: connect.session() MemoryStore is not
designed for a production environment, as it will leak
memory, and will not scale past a single process.
2019-05-17T07:52:09.001Z box:shell reload spawn: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/reloadnginx.sh
2019-05-17T07:52:09.020Z box:platform initializing addon infrastructure
2019-05-17T07:52:09.021Z box:platform Updating infrastructure from 48.12.1 to 48.14.0
2019-05-17T07:52:09.022Z box:locker Acquired : platform_start
2019-05-17T07:52:09.022Z box:platform stopping all containers for infra upgrade
2019-05-17T07:52:09.022Z box:shell stopContainers exec: docker ps -qa --filter 'network=cloudron' | xargs --no-run-if-empty docker stop
2019-05-17T07:52:09.048Z box:dockerproxy startDockerProxy: started proxy on port 3003
2019-05-17T07:52:09.048Z Cloudron is up and running
2019-05-17T07:52:09.064Z box:shell reload spawn: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/reloadnginx.sh
2019-05-17T07:52:09.480Z box:reverseproxy configureDefaultServer: done
2019-05-17T07:52:09.547Z box:shell stopContainers (stdout):
2019-05-17T07:52:09.547Z box:shell stopContainers (stderr):
2019-05-17T07:52:09.547Z box:shell stopContainers exec: docker ps -qa --filter 'network=cloudron' | xargs --no-run-if-empty docker rm -f
2019-05-17T07:52:09.602Z box:shell stopContainers (stdout):
2019-05-17T07:52:09.602Z box:shell stopContainers (stderr):
2019-05-17T07:52:09.602Z box:platform startApps: reconfiguring installed apps
2019-05-17T07:52:09.607Z box:cloudron AppsError: Internal error
    at /home/yellowtent/box/src/apps.js:1413:36
    at /home/yellowtent/box/src/apps.js:521:31
    at /home/yellowtent/box/src/appdb.js:243:27
    at Query.args.(anonymous function) (/home/yellowtent/box/src/database.js:154:9)
    at Query.<anonymous> (/home/yellowtent/box/node_modules/mysql/lib/Connection.js:525:10)
    at Query._callback (/home/yellowtent/box/node_modules/mysql/lib/Connection.js:491:16)
    at Query.Sequence.end (/home/yellowtent/box/node_modules/mysql/lib/protocol/sequences/Sequence.js:83:24)
    at Query.ErrorPacket (/home/yellowtent/box/node_modules/mysql/lib/protocol/sequences/Query.js:90:8)
    at Protocol._parsePacket (/home/yellowtent/box/node_modules/mysql/lib/protocol/Protocol.js:291:23)
    at Parser._parsePacket (/home/yellowtent/box/node_modules/mysql/lib/protocol/Parser.js:433:10)
2019-05-17T07:52:14.423Z ERROR Internal error [ /home/yellowtent/box/node_modules/connect-lastmile/lib/index.js:37:25 ]
stack:
  """
    HttpError: Internal error
        at /home/yellowtent/box/src/routes/apps.js:59:32
        at /home/yellowtent/box/src/apps.js:535:27
        at /home/yellowtent/box/src/apps.js:521:31
        at /home/yellowtent/box/src/appdb.js:243:27
        at Query.args.(anonymous function) (/home/yellowtent/box/src/database.js:154:9)
        at Query.<anonymous> (/home/yellowtent/box/node_modules/mysql/lib/Connection.js:525:10)
        at Query._callback (/home/yellowtent/box/node_modules/mysql/lib/Connection.js:491:16)
        at Query.Sequence.end (/home/yellowtent/box/node_modules/mysql/lib/protocol/sequences/Sequence.js:83:24)
        at Query.ErrorPacket (/home/yellowtent/box/node_modules/mysql/lib/protocol/sequences/Query.js:90:8)
        at Protocol._parsePacket (/home/yellowtent/box/node_modules/mysql/lib/protocol/Protocol.js:291:23)
  """
name:          HttpError
status:        500
message:       Internal error
internalError:
  """
    stack:
      """
        AppsError: Internal error
            at /home/yellowtent/box/src/apps.js:521:40
            at /home/yellowtent/box/src/appdb.js:243:27
            at Query.args.(anonymous function) (/home/yellowtent/box/src/database.js:154:9)
            at Query.<anonymous> (/home/yellowtent/box/node_modules/mysql/lib/Connection.js:525:10)
            at Query._callback (/home/yellowtent/box/node_modules/mysql/lib/Connection.js:491:16)
            at Query.Sequence.end (/home/yellowtent/box/node_modules/mysql/lib/protocol/sequences/Sequence.js:83:24)
            at Query.ErrorPacket (/home/yellowtent/box/node_modules/mysql/lib/protocol/sequences/Query.js:90:8)
            at Protocol._parsePacket (/home/yellowtent/box/node_modules/mysql/lib/protocol/Protocol.js:291:23)
            at Parser._parsePacket (/home/yellowtent/box/node_modules/mysql/lib/protocol/Parser.js:433:10)
            at Parser.write (/home/yellowtent/box/node_modules/mysql/lib/protocol/Parser.js:43:10)
      """
    name:        AppsError
    reason:      Internal Error
    message:     Internal error
    nestedError:
      """
        stack:
          """
            DatabaseError: Internal error
                at /home/yellowtent/box/src/appdb.js:243:36
                at Query.args.(anonymous function) (/home/yellowtent/box/src/database.js:154:9)
                at Query.<anonymous> (/home/yellowtent/box/node_modules/mysql/lib/Connection.js:525:10)
                at Query._callback (/home/yellowtent/box/node_modules/mysql/lib/Connection.js:491:16)
                at Query.Sequence.end (/home/yellowtent/box/node_modules/mysql/lib/protocol/sequences/Sequence.js:83:24)
                at Query.ErrorPacket (/home/yellowtent/box/node_modules/mysql/lib/protocol/sequences/Query.js:90:8)
                at Protocol._parsePacket (/home/yellowtent/box/node_modules/mysql/lib/protocol/Protocol.js:291:23)
                at Parser._parsePacket (/home/yellowtent/box/node_modules/mysql/lib/protocol/Parser.js:433:10)
                at Parser.write (/home/yellowtent/box/node_modules/mysql/lib/protocol/Parser.js:43:10)
                at Protocol.write (/home/yellowtent/box/node_modules/mysql/lib/protocol/Protocol.js:38:16)
          """
        name:        DatabaseError
        reason:      Internal error
        message:     Internal error
        nestedError:
          """
            stack:
              """
                Error: ER_BAD_FIELD_ERROR: Unknown column 'apps.label' in 'field list'
                    at Query.Sequence._packetToError (/home/yellowtent/box/node_modules/mysql/lib/protocol/sequences/Sequence.js:47:14)
                    at Query.ErrorPacket (/home/yellowtent/box/node_modules/mysql/lib/protocol/sequences/Query.js:77:18)
                    at Protocol._parsePacket (/home/yellowtent/box/node_modules/mysql/lib/protocol/Protocol.js:291:23)
                    at Parser._parsePacket (/home/yellowtent/box/node_modules/mysql/lib/protocol/Parser.js:433:10)
                    at Parser.write (/home/yellowtent/box/node_modules/mysql/lib/protocol/Parser.js:43:10)
                    at Protocol.write (/home/yellowtent/box/node_modules/mysql/lib/protocol/Protocol.js:38:16)
                    at Socket.<anonymous> (/home/yellowtent/box/node_modules/mysql/lib/Connection.js:91:28)
                    at Socket.<anonymous> (/home/yellowtent/box/node_modules/mysql/lib/Connection.js:525:10)
                    at Socket.emit (events.js:189:13)
                    at addChunk (_stream_readable.js:284:12)
                    --------------------
                    at Protocol._enqueue (/home/yellowtent/box/node_modules/mysql/lib/protocol/Protocol.js:144:48)
                    at PoolConnection.query (/home/yellowtent/box/node_modules/mysql/lib/Connection.js:201:25)
                    at Object.query (/home/yellowtent/box/src/database.js:157:30)
                    at Object.getAll (/home/yellowtent/box/src/appdb.js:235:14)
                    at /home/yellowtent/box/src/apps.js:520:15
                    at /home/yellowtent/box/src/apps.js:449:9
                    at /home/yellowtent/box/src/domaindb.js:52:9
                    at Query.args.(anonymous function) (/home/yellowtent/box/src/database.js:154:9)
                    at Query.<anonymous> (/home/yellowtent/box/node_modules/mysql/lib/Connection.js:525:10)
                    at Query._callback (/home/yellowtent/box/node_modules/mysql/lib/Connection.js:491:16)
              """
            message:    ER_BAD_FIELD_ERROR: Unknown column 'apps.label' in 'field list'
            code:       ER_BAD_FIELD_ERROR
            errno:      1054
            sqlMessage: Unknown column 'apps.label' in 'field list'
            sqlState:   42S22
            index:      0
            sql:        SELECT apps.id,apps.appStoreId,apps.installationState,apps.installationProgress,apps.runState,apps.health,apps.containerId,apps.manifestJson,apps.httpPort,subdomains.subdomain AS location,subdomains.domain,apps.accessRestrictionJson,apps.restoreConfigJson,apps
.oldConfigJson,apps.updateConfigJson,apps.memoryLimit,apps.label,apps.tagsJson,apps.xFrameOptions,apps.sso,apps.debugModeJson,apps.robotsTxt,apps.enableBackup,apps.creationTime,apps.updateTime,apps.ownerId,apps.mailboxName,apps.enableAutomaticUpdate,apps.dataDir,apps.ts,apps.healthTi
me,GROUP_CONCAT(CAST(appPortBindings.hostPort AS CHAR(6))) AS hostPorts, GROUP_CONCAT(appPortBindings.environmentVariable) AS environmentVariables, GROUP_CONCAT(appPortBindings.type) AS portTypes,JSON_ARRAYAGG(appEnvVars.name) AS envNames, JSON_ARRAYAGG(appEnvVars.value) AS envValues
 FROM apps  LEFT OUTER JOIN appPortBindings ON apps.id = appPortBindings.appId  LEFT OUTER JOIN appEnvVars ON apps.id = appEnvVars.appId  LEFT OUTER JOIN subdomains ON apps.id = subdomains.appId AND subdomains.type = 'primary' GROUP BY apps.id ORDER BY apps.id
          """
      """
  """
Box GET /api/v1/apps 500 6.846 ms - 70

I ran the script as root user. Not sudo.
The error after running the script is network related?

What am I doing wrong?

Thanks!
Florian

nebulon

To clarify, the error is that the database schema is not updated but the code is expecting the latest schema to be there.

Now the start.sh script would update the database schema alongside with restarting other services. At least the start.sh logs you have pasted there don't even show the db migration to be happening. You should see ==> Migrating data somewhere. The other error Error response from daemon: network with name cloudron already exists is only a warning, that the network already exists, this is fine and not an issue.
If possible, can you paste the whole output of the start.sh script?

root_iam

the above was the whole output of the script. I was really sure.

I tried again and the output was a lot bigger:

root@my:/home/yellowtent/box/setup# ./start.sh
==> Cloudron Start
==> Configuring docker
apparmor.service is not a native service, redirecting to systemd-sysv-install
Executing /lib/systemd/systemd-sysv-install enable apparmor
Error response from daemon: network with name cloudron already exists
==> Ensuring directories
==> Configuring journald
==> Setting up unbound
==> Adding systemd services
Synchronizing state of unbound.service with SysV init with /lib/systemd/systemd-sysv-install...
Executing /lib/systemd/systemd-sysv-install enable unbound
Synchronizing state of cron.service with SysV init with /lib/systemd/systemd-sysv-install...
Executing /lib/systemd/systemd-sysv-install enable cron
json: updated "/etc/cloudron/cloudron.conf" in-place
==> Clearing custom.yml
==> Configuring sudoers
==> Configuring collectd
==> Configuring logrotate
==> Adding motd message for admins
==> Configuring nginx
Job for mysql.service failed because the control process exited with error code. See "systemctl status mysql.service" and "journalctl -xe" for details.
Restarting MySql again after sometime since this fails randomly
Job for mysql.service failed because the control process exited with error code. See "systemctl status mysql.service" and "journalctl -xe" for details.
Restarting MySql again after sometime since this fails randomly
Job for mysql.service failed because the control process exited with error code. See "systemctl status mysql.service" and "journalctl -xe" for details.
Restarting MySql again after sometime since this fails randomly
Job for mysql.service failed because the control process exited with error code. See "systemctl status mysql.service" and "journalctl -xe" for details.
Restarting MySql again after sometime since this fails randomly
Job for mysql.service failed because the control process exited with error code. See "systemctl status mysql.service" and "journalctl -xe" for details.
Restarting MySql again after sometime since this fails randomly
Job for mysql.service failed because the control process exited with error code. See "systemctl status mysql.service" and "journalctl -xe" for details.
Restarting MySql again after sometime since this fails randomly

The last part was a loop. Stopped it aftere several times.

Here the output of "systemctl status mysql.service"

root@my:/home/yellowtent/box/setup# systemctl status mysql.service
● mysql.service - MySQL Community Server
   Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
   Active: activating (start-post) (Result: exit-code) since Sat 2019-05-18 10:53:52 UTC; 20s ago
  Process: 31446 ExecStart=/usr/sbin/mysqld (code=exited, status=1/FAILURE)
  Process: 31437 ExecStartPre=/usr/share/mysql/mysql-systemd-start pre (code=exited, status=0/SUCCESS)
 Main PID: 31446 (code=exited, status=1/FAILURE);         : 31447 (mysql-systemd-s)
    Tasks: 2
   Memory: 304.0K
      CPU: 479ms
   CGroup: /system.slice/mysql.service
           └─control
             ├─31447 /bin/bash /usr/share/mysql/mysql-systemd-start post
             └─31502 sleep 1

May 18 10:53:52 *** systemd[1]: Starting MySQL Community Server...
May 18 10:53:53 *** systemd[1]: mysql.service: Main process exited, code=exited, status=1/FAILURE

And the output of "journalctl -xe"

root@my:/home/yellowtent/box/setup# journalctl -xe
May 18 10:54:23 *** systemd[1]: mysql.service: Service hold-off time over, scheduling restart.
May 18 10:54:23 *** systemd[1]: Stopped MySQL Community Server.
-- Subject: Unit mysql.service has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit mysql.service has finished shutting down.
May 18 10:54:23 *** systemd[1]: Starting MySQL Community Server...
-- Subject: Unit mysql.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit mysql.service has begun starting up.
May 18 10:54:23 *** audit[31539]: AVC apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/proc/31539/status" pid=31539 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=112 ouid=112
May 18 10:54:23 *** audit[31539]: AVC apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/sys/devices/system/node/" pid=31539 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=112 ouid=0
May 18 10:54:23 *** kernel: audit: type=1400 audit(1558176863.653:93): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/proc/31539/status" pid=31539 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=112 ouid=112
May 18 10:54:23 *** kernel: audit: type=1400 audit(1558176863.653:94): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/sys/devices/system/node/" pid=31539 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=112 ouid=0
May 18 10:54:23 *** kernel: audit: type=1400 audit(1558176863.653:95): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/proc/31539/status" pid=31539 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=112 ouid=112
May 18 10:54:23 *** audit[31539]: AVC apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/proc/31539/status" pid=31539 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=112 ouid=112
May 18 10:54:23 *** sshd[31523]: Failed password for invalid user cstrike from 51.38.90.195 port 56252 ssh2
May 18 10:54:23 *** sshd[31523]: Received disconnect from 51.38.90.195 port 56252:11: Bye Bye [preauth]
May 18 10:54:23 *** sshd[31523]: Disconnected from 51.38.90.195 port 56252 [preauth]
May 18 10:54:24 *** systemd[1]: mysql.service: Main process exited, code=exited, status=1/FAILURE
May 18 10:54:28 *** sshd[31567]: Invalid user bouncer from 51.75.204.92
May 18 10:54:28 *** sshd[31567]: input_userauth_request: invalid user bouncer [preauth]
May 18 10:54:28 *** sshd[31567]: pam_unix(sshd:auth): check pass; user unknown
May 18 10:54:28 *** sshd[31567]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=51.75.204.92
May 18 10:54:29 *** sshd[31566]: Invalid user stage from 187.190.45.111
May 18 10:54:29 *** sshd[31566]: input_userauth_request: invalid user stage [preauth]
May 18 10:54:29 *** sshd[31566]: pam_unix(sshd:auth): check pass; user unknown
May 18 10:54:29 *** sshd[31566]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=187.190.45.111
May 18 10:54:30 *** sshd[31566]: Failed password for invalid user stage from 187.190.45.111 port 56888 ssh2
May 18 10:54:30 *** sshd[31567]: Failed password for invalid user bouncer from 51.75.204.92 port 42732 ssh2
May 18 10:54:30 *** sshd[31567]: Received disconnect from 51.75.204.92 port 42732:11: Bye Bye [preauth]
May 18 10:54:30 *** sshd[31567]: Disconnected from 51.75.204.92 port 42732 [preauth]
May 18 10:54:30 *** sshd[31566]: Received disconnect from 187.190.45.111 port 56888:11: Bye Bye [preauth]
May 18 10:54:30 *** sshd[31566]: Disconnected from 187.190.45.111 port 56888 [preauth]
May 18 10:54:35 *** sshd[31576]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=218.92.0.205  user=root
May 18 10:54:36 *** sshd[31576]: Failed password for root from 218.92.0.205 port 26892 ssh2
May 18 10:54:39 *** sshd[31576]: Failed password for root from 218.92.0.205 port 26892 ssh2
May 18 10:54:42 *** sshd[31576]: Failed password for root from 218.92.0.205 port 26892 ssh2
May 18 10:54:42 *** sshd[31576]: Received disconnect from 218.92.0.205 port 26892:11:  [preauth]
May 18 10:54:42 *** sshd[31576]: Disconnected from 218.92.0.205 port 26892 [preauth]
May 18 10:54:42 *** sshd[31576]: PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=218.92.0.205  user=root
May 18 10:54:53 *** systemd[1]: Failed to start MySQL Community Server.
-- Subject: Unit mysql.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit mysql.service has failed.
--
-- The result is failed.
May 18 10:54:53 *** systemd[1]: mysql.service: Unit entered failed state.
May 18 10:54:53 *** systemd[1]: mysql.service: Failed with result 'exit-code'.
May 18 10:54:54 *** systemd[1]: mysql.service: Service hold-off time over, scheduling restart.
May 18 10:54:54 *** systemd[1]: Stopped MySQL Community Server.
-- Subject: Unit mysql.service has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit mysql.service has finished shutting down.
May 18 10:54:54 *** systemd[1]: Starting MySQL Community Server...
-- Subject: Unit mysql.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit mysql.service has begun starting up.
May 18 10:54:54 *** audit[31634]: AVC apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/proc/31634/status" pid=31634 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=112 ouid=112
May 18 10:54:54 *** audit[31634]: AVC apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/sys/devices/system/node/" pid=31634 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=112 ouid=0
May 18 10:54:54 *** kernel: audit: type=1400 audit(1558176894.209:96): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/proc/31634/status" pid=31634 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=112 ouid=112
May 18 10:54:54 *** kernel: audit: type=1400 audit(1558176894.209:97): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/sys/devices/system/node/" pid=31634 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=112 ouid=0
May 18 10:54:54 *** kernel: audit: type=1400 audit(1558176894.209:98): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/proc/31634/status" pid=31634 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=112 ouid=112
May 18 10:54:54 *** audit[31634]: AVC apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/proc/31634/status" pid=31634 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=112 ouid=112
May 18 10:54:55 *** systemd[1]: mysql.service: Main process exited, code=exited, status=1/FAILURE
May 18 10:54:57 *** sshd[31658]: Invalid user zabbix from 179.157.8.130
May 18 10:54:57 *** sshd[31658]: input_userauth_request: invalid user zabbix [preauth]
May 18 10:54:57 *** sshd[31658]: pam_unix(sshd:auth): check pass; user unknown
May 18 10:54:57 *** sshd[31658]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=179.157.8.130
May 18 10:54:58 *** sshd[31658]: Failed password for invalid user zabbix from 179.157.8.130 port 49808 ssh2
May 18 10:54:59 *** sshd[31658]: Received disconnect from 179.157.8.130 port 49808:11: Bye Bye [preauth]
May 18 10:54:59 *** sshd[31658]: Disconnected from 179.157.8.130 port 49808 [preauth]

Looks like a password problem?

root_iam

Problem solved!

After several loops of booting and running the script the migration seemed to work.
Cloudron is up as are all apps!

Thanks nebulon!!

skiqh

@nebulon Thank you very much! I had a similar problem, probably also originating in an incomplete schema migration — the error message I got was ER_BAD_FIELD_ERROR: Unknown column 'apps.reverseProxyConfigJson' in 'field list'(putting it here so anyone encountering the same may find it) — anyway, running the command /home/yellowtent/box/setup/start.sh as root just as you suggested fixed the problem at once, even without a reboot.

Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.

Cloudron Forum

mysql error: ER_BAD_FIELD_ERROR: Unknown column 'apps.label' in 'field list'