LDAP Authentication Failing Everywhere

[eganonoa]

Last night I ran the system update to the latest version of Cloudron. Today LDAP authentication is totally broken. I can access and log into any app that isn't linked to the Cloudron LDAP, but anything that is (including critically the main cloudron app, my.xxxx.xxx) cannot be accessed at all. Those apps like Nextcloud that have both an admin user not attached to Cloudron's LDAP and users associated with Cloudron's LDAP can only work for those not-attached to the Cloudron LDAP. So it seems that the problem is the LDAP.

Unfortunately I cannot troubleshoot this at all, because I have no idea even how Cloudron provides the LDAP. It doesn't appear to be in its own container. But similarly I can't seem to find it as a systemd service.

Any help would be very much appreciated, even just helping me locate where the LDAP server is!

Thanks

[nebulon]

@eganonoa can you check the logs at /home/yellowtent/platformdata/logs/box.log about any errors during login? Also you can try to restart the box service with systemctl restart box and see if that just somewhat got stuck? This process would handle the LDAP queries from the apps.

[eganonoa]

@nebulon Thanks for getting back to me. So many errors it's hard to know what's going on. There is one error that is repeated constantly:

Box GET /api/v1/apps 500 Internal Server Error ER_BAD_FIELD_ERROR: Unknown column 'inviteToken'
 in 'field list' 3.242 ms - 120
HttpError: ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list'
    at tokenAuth (/home/yellowtent/box/src/routes/accesscontrol.js:74:28)
    at processTicksAndRejections (internal/process/task_queues.js:95:5) {
  status: 500,
  internalError: null,
  details: null
}

[eganonoa]

@nebulon In addition to the error above, I'm seeing repeated errors like the following indicating that it cannot remove unused images of various service containers (the one below is redis, but there are many) because the image is in use by a running container.

2021-11-10T12:58:41.101Z box:platform pruneInfraImages: removing unused image of cloudron/redis
: tag: cloudron/redis:3.0.3@sha256:37e5222e01ae89bc5a742ce12030631de25a127b5deec8a0e992c68df0fd
ec10 id: 0d0ead1c7434
Error response from daemon: conflict: unable to delete 0d0ead1c7434 (cannot be forced) - image
is being used by running container 63047d0c039d

[eganonoa]

@nebulon To give you the full flavor, this is what it looks like in the first couple of minutes after restarting box or restarting the server as a whole:

2021-11-10T13:25:00.744Z box:server ==========================================
2021-11-10T13:25:00.745Z box:server            Cloudron 7.0.3
2021-11-10T13:25:00.745Z box:server ==========================================
2021-11-10T13:25:00.795Z box:settings initCache: pre-load settings
2021-11-10T13:25:00.818Z box:tasks stopAllTasks: stopping all tasks
2021-11-10T13:25:00.819Z box:shell stopTask spawn: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/stoptask.sh all
2021-11-10T13:25:00.831Z box:shell stopTask (stdout): sudo: unable to resolve host cloudron: Name or service not known

2021-11-10T13:25:00.914Z box:shell removeCollectdProfile spawn: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/configurecollectd.sh remove cloudron-backup
2021-11-10T13:25:00.924Z box:shell removeCollectdProfile (stdout): sudo: unable to resolve host cloudron: Name or service not known

2021-11-10T13:25:00.928Z box:shell removeCollectdProfile (stdout): Restarting collectd

2021-11-10T13:25:00.990Z box:shell removeCollectdProfile (stdout): Removing collectd stats of cloudron-backup

2021-11-10T13:25:00.992Z box:reverseproxy writeDashboardConfig: writing admin config for unpo.org
2021-11-10T13:25:01.015Z box:shell reload spawn: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/restartservice.sh nginx
2021-11-10T13:25:01.023Z box:shell reload (stdout): sudo: unable to resolve host cloudron: Name or service not known

2021-11-10T13:25:01.110Z box:cloudron onActivated: running post activation tasks
2021-11-10T13:25:01.110Z box:platform initializing addon infrastructure
2021-11-10T13:25:01.111Z box:platform Updating infrastructure from 48.20.0 to 48.20.0
2021-11-10T13:25:01.112Z box:locker Acquired : platform_start
2021-11-10T13:25:01.112Z box:platform markApps: changedAddons: ["postgresql","redis"]
2021-11-10T13:25:01.117Z box:platform Failed to start services. retry=false (attempt 0): ER_BAD_FIELD_ERROR: Unknown column 'apps.operatorsJson' in 'field list'
2021-11-10T13:25:01.117Z box:locker Released : platform_start
2021-11-10T13:25:01.117Z box:platform onPlatformReady: platform is ready. infra changed: true
2021-11-10T13:25:01.117Z box:platform pruneInfraImages: checking existing images
2021-11-10T13:25:01.425Z box:platform pruneInfraImages: removing unused image of cloudron/postgresql: tag: cloudron/postgresql:4.0.6@sha256:e583082e15e8e41b0e3b80c3efc917ec429f19fa08a19e14fc27144a8bfe446a id: fce82a9329ea
Error response from daemon: conflict: unable to delete fce82a9329ea (cannot be forced) - image is being used by running container eca761eafb80
2021-11-10T13:25:01.474Z box:platform Error removing image fce82a9329ea: undefined
2021-11-10T13:25:01.601Z box:platform pruneInfraImages: removing unused image of cloudron/redis: tag: cloudron/redis:3.0.3@sha256:37e5222e01ae89bc5a742ce12030631de25a127b5deec8a0e992c68df0fdec10 id: 0d0ead1c7434
Error response from daemon: conflict: unable to delete 0d0ead1c7434 (cannot be forced) - image is being used by running container b17c7b76bc11
2021-11-10T13:25:01.654Z box:platform Error removing image 0d0ead1c7434: undefined
2021-11-10T13:25:01.721Z box:platform pruneInfraImages: removing unused image of cloudron/mail: tag: cloudron/mail:3.3.3@sha256:b1093e6f38bebf4a9ae903ca385aea3a32e7cccae5ede7f2e01a34681e361a5f id: c09c98bad2da
Error response from daemon: conflict: unable to delete c09c98bad2da (cannot be forced) - image is being used by running container 1be6e3b10aef
2021-11-10T13:25:01.779Z box:platform Error removing image c09c98bad2da: undefined
2021-11-10T13:25:01.913Z box:platform pruneInfraImages: removing unused image of cloudron/sftp: tag: cloudron/sftp:3.3.0@sha256:183c11150d5a681cb02f7d2bd542ddb8a8f097422feafb7fac8fdbca0ca55d47 id: b7cf3b4d8551
Error response from daemon: conflict: unable to delete b7cf3b4d8551 (cannot be forced) - image is being used by running container 0ea0db968d74
2021-11-10T13:25:01.961Z box:platform Error removing image b7cf3b4d8551: undefined
2021-11-10T13:25:01.961Z box:apps schedulePendingTasks: scheduling app tasks
2021-11-10T13:25:01.969Z box:dockerproxy startDockerProxy: started proxy on port 3003
Cloudron is up and running. Logs are at /home/yellowtent/platformdata/logs/box.log
2021-11-10T13:25:01.980Z box:cloudron Startup task at index 3 failed: ER_BAD_FIELD_ERROR: Unknown column 'apps.operatorsJson' in 'field list'
HttpError: ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list'
    at tokenAuth (/home/yellowtent/box/src/routes/accesscontrol.js:74:28)
    at processTicksAndRejections (internal/process/task_queues.js:95:5) {
  status: 500,
  internalError: null,
  details: null
}
Box GET /api/v1/apps 500 Internal Server Error ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list' 15.072 ms - 120
HttpError: ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list'
    at tokenAuth (/home/yellowtent/box/src/routes/accesscontrol.js:74:28)
    at processTicksAndRejections (internal/process/task_queues.js:95:5) {
  status: 500,
  internalError: null,
  details: null
}
Box GET /api/v1/apps 500 Internal Server Error ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list' 3.535 ms - 120
HttpError: ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list'
    at tokenAuth (/home/yellowtent/box/src/routes/accesscontrol.js:74:28)
    at processTicksAndRejections (internal/process/task_queues.js:95:5) {
  status: 500,
  internalError: null,
  details: null
}
Box GET /api/v1/apps 500 Internal Server Error ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list' 3.470 ms - 120
HttpError: ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list'
    at tokenAuth (/home/yellowtent/box/src/routes/accesscontrol.js:74:28)
    at processTicksAndRejections (internal/process/task_queues.js:95:5) {
  status: 500,
  internalError: null,
  details: null
}
Box GET /api/v1/apps 500 Internal Server Error ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list' 5.179 ms - 120
HttpError: ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list'
    at tokenAuth (/home/yellowtent/box/src/routes/accesscontrol.js:74:28)
    at processTicksAndRejections (internal/process/task_queues.js:95:5) {
  status: 500,
  internalError: null,
  details: null
}
Box GET /api/v1/apps 500 Internal Server Error ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list' 5.143 ms - 120
HttpError: ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list'
    at tokenAuth (/home/yellowtent/box/src/routes/accesscontrol.js:74:28)
    at processTicksAndRejections (internal/process/task_queues.js:95:5) {
  status: 500,
  internalError: null,
  details: null
}
Box GET /api/v1/apps 500 Internal Server Error ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list' 3.395 ms - 120
HttpError: ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list'
    at tokenAuth (/home/yellowtent/box/src/routes/accesscontrol.js:74:28)
    at processTicksAndRejections (internal/process/task_queues.js:95:5) {
  status: 500,
  internalError: null,
  details: null
}
Box GET /api/v1/apps 500 Internal Server Error ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list' 4.189 ms - 120
HttpError: ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list'
    at tokenAuth (/home/yellowtent/box/src/routes/accesscontrol.js:74:28)
    at processTicksAndRejections (internal/process/task_queues.js:95:5) {
  status: 500,
  internalError: null,
  details: null
}
Box GET /api/v1/apps 500 Internal Server Error ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list' 3.155 ms - 120
HttpError: ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list'
    at tokenAuth (/home/yellowtent/box/src/routes/accesscontrol.js:74:28)
    at processTicksAndRejections (internal/process/task_queues.js:95:5) {
  status: 500,
  internalError: null,
  details: null
}
Box GET /api/v1/apps 500 Internal Server Error ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list' 3.152 ms - 120
HttpError: ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list'
    at tokenAuth (/home/yellowtent/box/src/routes/accesscontrol.js:74:28)
    at processTicksAndRejections (internal/process/task_queues.js:95:5) {
  status: 500,
  internalError: null,
  details: null
}
Box GET /api/v1/apps 500 Internal Server Error ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list' 3.424 ms - 120
HttpError: ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list'
    at tokenAuth (/home/yellowtent/box/src/routes/accesscontrol.js:74:28)
    at processTicksAndRejections (internal/process/task_queues.js:95:5) {
  status: 500,
  internalError: null,
  details: null
}
Box GET /api/v1/apps 500 Internal Server Error ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list' 3.551 ms - 120
HttpError: ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list'
    at tokenAuth (/home/yellowtent/box/src/routes/accesscontrol.js:74:28)
    at processTicksAndRejections (internal/process/task_queues.js:95:5) {
  status: 500,
  internalError: null,
  details: null
}
Box GET /api/v1/apps 500 Internal Server Error ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list' 3.437 ms - 120
HttpError: ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list'
    at tokenAuth (/home/yellowtent/box/src/routes/accesscontrol.js:74:28)
    at processTicksAndRejections (internal/process/task_queues.js:95:5) {
  status: 500,
  internalError: null,
  details: null
}
Box GET /api/v1/apps 500 Internal Server Error ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list' 3.121 ms - 120
HttpError: ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list'
    at tokenAuth (/home/yellowtent/box/src/routes/accesscontrol.js:74:28)
    at processTicksAndRejections (internal/process/task_queues.js:95:5) {
  status: 500,
  internalError: null,
  details: null
}
Box GET /api/v1/apps 500 Internal Server Error ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list' 4.228 ms - 120
HttpError: ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list'
    at tokenAuth (/home/yellowtent/box/src/routes/accesscontrol.js:74:28)
    at processTicksAndRejections (internal/process/task_queues.js:95:5) {
  status: 500,
  internalError: null,
  details: null
}
Box GET /api/v1/apps 500 Internal Server Error ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list' 4.359 ms - 120
HttpError: ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list'
    at tokenAuth (/home/yellowtent/box/src/routes/accesscontrol.js:74:28)
    at processTicksAndRejections (internal/process/task_queues.js:95:5) {
  status: 500,
  internalError: null,
  details: null
}
Box GET /api/v1/apps 500 Internal Server Error ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list' 3.248 ms - 120
HttpError: ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list'
    at tokenAuth (/home/yellowtent/box/src/routes/accesscontrol.js:74:28)
    at processTicksAndRejections (internal/process/task_queues.js:95:5) {
  status: 500,
  internalError: null,
  details: null
}
Box GET /api/v1/apps 500 Internal Server Error ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list' 4.106 ms - 120
HttpError: ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list'
    at tokenAuth (/home/yellowtent/box/src/routes/accesscontrol.js:74:28)
    at processTicksAndRejections (internal/process/task_queues.js:95:5) {
  status: 500,
  internalError: null,
  details: null
}
Box GET /api/v1/apps 500 Internal Server Error ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list' 3.918 ms - 120
HttpError: ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list'
    at tokenAuth (/home/yellowtent/box/src/routes/accesscontrol.js:74:28)
    at processTicksAndRejections (internal/process/task_queues.js:95:5) {
  status: 500,
  internalError: null,
  details: null
}
Box GET /api/v1/apps 500 Internal Server Error ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list' 3.196 ms - 120
HttpError: ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list'
    at tokenAuth (/home/yellowtent/box/src/routes/accesscontrol.js:74:28)
    at processTicksAndRejections (internal/process/task_queues.js:95:5) {
  status: 500,
  internalError: null,
  details: null
}
Box GET /api/v1/apps 500 Internal Server Error ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list' 2.925 ms - 120
HttpError: ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list'
    at tokenAuth (/home/yellowtent/box/src/routes/accesscontrol.js:74:28)
    at processTicksAndRejections (internal/process/task_queues.js:95:5) {
  status: 500,
  internalError: null,
  details: null
}
Box GET /api/v1/apps 500 Internal Server Error ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list' 3.497 ms - 120
HttpError: ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list'
    at tokenAuth (/home/yellowtent/box/src/routes/accesscontrol.js:74:28)
    at processTicksAndRejections (internal/process/task_queues.js:95:5) {
  status: 500,
  internalError: null,
  details: null
}
Box GET /api/v1/apps 500 Internal Server Error ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list' 3.649 ms - 120
HttpError: ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list'
    at tokenAuth (/home/yellowtent/box/src/routes/accesscontrol.js:74:28)
    at processTicksAndRejections (internal/process/task_queues.js:95:5) {
  status: 500,
  internalError: null,
  details: null
}
Box GET /api/v1/apps 500 Internal Server Error ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list' 3.752 ms - 120
HttpError: ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list'
    at tokenAuth (/home/yellowtent/box/src/routes/accesscontrol.js:74:28)
    at processTicksAndRejections (internal/process/task_queues.js:95:5) {
  status: 500,
  internalError: null,
  details: null
}
Box GET /api/v1/apps 500 Internal Server Error ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list' 4.672 ms - 120
HttpError: ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list'
    at tokenAuth (/home/yellowtent/box/src/routes/accesscontrol.js:74:28)
    at processTicksAndRejections (internal/process/task_queues.js:95:5) {
  status: 500,
  internalError: null,
  details: null
}
Box GET /api/v1/apps 500 Internal Server Error ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list' 2.939 ms - 120
HttpError: ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list'
    at tokenAuth (/home/yellowtent/box/src/routes/accesscontrol.js:74:28)
    at processTicksAndRejections (internal/process/task_queues.js:95:5) {
  status: 500,
  internalError: null,
  details: null
}
Box GET /api/v1/apps 500 Internal Server Error ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list' 3.461 ms - 120
HttpError: ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list'
    at tokenAuth (/home/yellowtent/box/src/routes/accesscontrol.js:74:28)
    at processTicksAndRejections (internal/process/task_queues.js:95:5) {
  status: 500,
  internalError: null,
  details: null
}
Box GET /api/v1/apps 500 Internal Server Error ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list' 3.180 ms - 120
HttpError: ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list'
    at tokenAuth (/home/yellowtent/box/src/routes/accesscontrol.js:74:28)
    at processTicksAndRejections (internal/process/task_queues.js:95:5) {
  status: 500,
  internalError: null,
  details: null
}
Box GET /api/v1/apps 500 Internal Server Error ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list' 3.825 ms - 120
HttpError: ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list'
    at tokenAuth (/home/yellowtent/box/src/routes/accesscontrol.js:74:28)
    at processTicksAndRejections (internal/process/task_queues.js:95:5) {
  status: 500,
  internalError: null,
  details: null
}
Box GET /api/v1/apps 500 Internal Server Error ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list' 3.037 ms - 120
HttpError: ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list'
    at tokenAuth (/home/yellowtent/box/src/routes/accesscontrol.js:74:28)
    at processTicksAndRejections (internal/process/task_queues.js:95:5) {
  status: 500,
  internalError: null,
  details: null
}
Box GET /api/v1/apps 500 Internal Server Error ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list' 2.803 ms - 120
HttpError: ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list'
    at tokenAuth (/home/yellowtent/box/src/routes/accesscontrol.js:74:28)
    at processTicksAndRejections (internal/process/task_queues.js:95:5) {
  status: 500,
  internalError: null,
  details: null
}
Box GET /api/v1/apps 500 Internal Server Error ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list' 2.988 ms - 120
HttpError: ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list'
    at tokenAuth (/home/yellowtent/box/src/routes/accesscontrol.js:74:28)
    at processTicksAndRejections (internal/process/task_queues.js:95:5) {
  status: 500,
  internalError: null,
  details: null
}
Box GET /api/v1/apps 500 Internal Server Error ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list' 2.850 ms - 120
HttpError: ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list'
    at tokenAuth (/home/yellowtent/box/src/routes/accesscontrol.js:74:28)
    at processTicksAndRejections (internal/process/task_queues.js:95:5) {
  status: 500,
  internalError: null,
  details: null
}
Box GET /api/v1/apps 500 Internal Server Error ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list' 2.985 ms - 120
HttpError: ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list'
    at tokenAuth (/home/yellowtent/box/src/routes/accesscontrol.js:74:28)
    at processTicksAndRejections (internal/process/task_queues.js:95:5) {
  status: 500,
  internalError: null,
  details: null
}
Box GET /api/v1/apps 500 Internal Server Error ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list' 2.830 ms - 120
HttpError: ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list'
    at tokenAuth (/home/yellowtent/box/src/routes/accesscontrol.js:74:28)
    at processTicksAndRejections (internal/process/task_queues.js:95:5) {
  status: 500,
  internalError: null,
  details: null
}
Box GET /api/v1/apps 500 Internal Server Error ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list' 2.931 ms - 120
HttpError: ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list'
    at tokenAuth (/home/yellowtent/box/src/routes/accesscontrol.js:74:28)
    at processTicksAndRejections (internal/process/task_queues.js:95:5) {
  status: 500,
  internalError: null,
  details: null
}
Box GET /api/v1/apps 500 Internal Server Error ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list' 3.424 ms - 120
HttpError: ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list'
    at tokenAuth (/home/yellowtent/box/src/routes/accesscontrol.js:74:28)
    at processTicksAndRejections (internal/process/task_queues.js:95:5) {
  status: 500,
  internalError: null,
  details: null
}
Box GET /api/v1/apps 500 Internal Server Error ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list' 4.224 ms - 120
HttpError: ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list'
    at tokenAuth (/home/yellowtent/box/src/routes/accesscontrol.js:74:28)
    at processTicksAndRejections (internal/process/task_queues.js:95:5) {
  status: 500,
  internalError: null,
  details: null
}
Box GET /api/v1/apps 500 Internal Server Error ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list' 3.103 ms - 120
HttpError: ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list'
    at tokenAuth (/home/yellowtent/box/src/routes/accesscontrol.js:74:28)
    at processTicksAndRejections (internal/process/task_queues.js:95:5) {
  status: 500,
  internalError: null,
  details: null
}
Box GET /api/v1/apps 500 Internal Server Error ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list' 2.951 ms - 120
HttpError: ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list'
    at tokenAuth (/home/yellowtent/box/src/routes/accesscontrol.js:74:28)
    at processTicksAndRejections (internal/process/task_queues.js:95:5) {
  status: 500,
  internalError: null,
  details: null
}
Box GET /api/v1/apps 500 Internal Server Error ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list' 3.464 ms - 120
HttpError: ER_BAD_FIELD_ERROR: Unknown column 'inviteToken' in 'field list'
    at tokenAuth (/home/yellowtent/box/src/routes/accesscontrol.js:74:28)
    at processTicksAndRejections (internal/process/task_queues.js:95:5) {
  status: 500,
  internalError: null,
  details: null
}

[nebulon]

Looks like the database migration failed midway then. Can you run /home/yellowtent/box/setup/start.sh as root on the server?

[eganonoa]

@nebulon That did it! Thank you.

Has been a strange update. I was on the latest version 6 before. This happened. And also Greenlight appears to have been rendered unusable. I'll post something about that also.

[girish]

@eganonoa which Ubuntu version are you on? If it's 16.04, there are some strange docker issues in that indeed. Please upgrade to Ubuntu 18

[eganonoa]

@girish Thanks, I'm on 20.04. The update was super-weird, but everything (inc. greenlight) now resolved. Greenlight was extremely odd. Just didn't work at all, even after the fix above. But I put it in recovery mode, it failed to start, but then after reboot of the server it worked again.

[girish]

To follow up this one, on some servers we see stream copy error: reading from a closed fifo in journalctl -u docker. When this happens, docker fails for some commands:

# docker restart ed925536-0e71-4646-95b6-3cdb99174e02
Error response from daemon: Cannot restart container ed925536-0e71-4646-95b6-3cdb99174e02: task ea04e90bc7c2fe012f2c85b6c88837a580cd4ccd0a0c960ca81955f104280e4a already exists: unknown

The upstream issue is https://github.com/docker/for-linux/issues/402 and https://github.com/docker/for-linux/issues/1062 which has no fix yet.

[girish]

Restarting docker does not make that error go away It seems the only way is to delete the container by hand and then Repair works. So:

docker rm -f ed925536-0e71-4646-95b6-3cdb99174e02

[girish]

If you hit this error, you will see something like below in the UI:

Docker Error: (HTTP code 500) server error - Cannot restart container 9e3dcc02-5547-4e4f-b914-78055b8eed8c: task 8830ba5131609cf8fbb82727901c2903e3b45d21666714d0d4243ac465c24d5b already exists: unknown

To fix:

• First, Repair -> Enable recovery mode.
• Then, Repair -> Disable recovery mode.

Internally, this just re-creates the container and that seems to fix it.

[robi]

I've had this come up as well.

An app restart fails the first time as well. The retry-task eventually succeeds, so what @girish said is a faster way with less clicks.