"Error" for multiple apps



  • Multiple apps on my Cloudron (version 2.3.2) are in "Error" status.
    Accessing the apps throw an nginx 404 page.

    Before I discovered this situation, I did

    apt update 
    apt autoremove
    dpkg --configure --a
    reboot
    

    because the admin panel was stuck at an automaticbox update, and/boot was nearly full due to kernel compilations.
    However, I'm not sure if the problem already existed beforehand (or not).

    Attempting to solve the problem, I tried to

    • repair the app
    • restore from the latest backup
    • restore from earlier backups
    • uninstall the app
      but none of the above worked and ended with "Error".

    I also attempted

    • installing an new instance of one of the faulty apps -> "Error"
    • installing a new instance of an app that's not one of the faulty ones -> "Error"
    • Clicking "Backup now" in settings -> "external error"

    Edit: now I'm pretty sure I accessed the admin panel yesterday and things seemed fine. so the problem's prob due to

    • fiddling with apt and dpkg manually
    • the update got aborted midway by reboot and now I end up with a broken installation


  • Hey, what worked for me after /boot was 100% full and the update-installer was stuck:

    //show all kernels not in use
    sudo dpkg --list 'linux-image*'|awk '{ if ($1=="ii") print $2}'|grep -v `uname -r`
    sudo systemctl stop box
    sudo systemctl stop docker
    //do ps -ax to find the id of the installer
    sudo kill -9 ID_of_installer.sh
    //remove old kernels, adjust this to the results from above
    sudo rm -rf /boot/*-4.4.0-{98,97,96,93,62}-*
    //fix packages
    sudo apt-get -f install
    //automatically remove unneeded kernels
    sudo purge-old-kernels
    //bring it back on
    sudo systemctl restart box
    sudo systemctl restart docker
    sudo systemctl restart cloudron.target

    edit: hope i didn't forget to escape any ' or *



  • Thanks @msbt for the your answer.
    sudo apt-get -f install doesn't (re)install anything...

    service box status gives some errors (partly redacted)

      node[2046]: box:apphealthmonitor [domain] org.ghost.cloudronapp2 skipped. istate:error rstate:stopped - 
      node[2046]: box:apphealthmonitor apps alive: [naked_domain|com.nextcloud.cloudronapp, note|de.nebulon.guacamoly, git|com.gitlab.cloudronapp, ...
      node[2046]: box:apphealthmonitor [domain] org.etherpad.cloudronapp skipped. istate:error rstate:stopped - 
      node[2046]: box:apphealthmonitor [domain] com.monicahq.cloudronapp skipped. istate:error rstate:stopped - 
      node[2046]: box:apphealthmonitor [domain] com.invoiceninja.cloudronapp skipped. istate:error rstate:stopped - 
      node[2046]: box:apphealthmonitor [domain] org.wordpress.cloudronapp skipped. istate:error rstate:stopped - 
      node[2046]: box:apphealthmonitor [domain] org.tt_rss.cloudronapp skipped. istate:error rstate:stopped - 
      node[2046]: box:apphealthmonitor [domain] org.ghost.cloudronapp2 skipped. istate:error rstate:stopped - 
      node[2046]: box:apphealthmonitor [domain] org.ghost.cloudronapp2 skipped. istate:error rstate:stopped - 
      node[2046]: box:apphealthmonitor apps alive: [naked_domain|com.nextcloud.cloudronapp, note|de.nebulon.guacamoly, git|com.gitlab.cloudronapp, ...
    

    Please kindly note that I'm not having problems with /boot anymore nor am I trying to upgrade - /boot now has some free space and apt upgrade has cleared...



  • I'm wondering if the system is missing an necessary kernal...
    I now have 4.4.0-127 and 4.4.0-128

    $ apt list linux-image* --installed
    Listing... Done
    linux-image-4.4.0-127-generic/xenial-updates,xenial-security,now 4.4.0-127.153 amd64 [installed,automatic]
    linux-image-4.4.0-128-generic/xenial-updates,xenial-security,now 4.4.0-128.154 amd64 [installed,automatic]
    linux-image-extra-4.4.0-127-generic/xenial-updates,xenial-security,now 4.4.0-127.153 amd64 [installed,automatic]
    linux-image-extra-4.4.0-128-generic/xenial-updates,xenial-security,now 4.4.0-128.154 amd64 [installed,automatic]
    


  • @uiharu if you cleared the boot issue, just press configure button of an app and press save. The app should come back up.

    If not, can you paste the output of 'docker ps'?



  • Thanks @girish

    I tried again but still get "Error"

    • repair the app
    • restore from the latest backup
    • restore from earlier backups
    • uninstall the app
      but none of the above worked and ended with "Error".

    I also attempted

    • installing an new instance of one of the faulty apps -> "Error"
    • installing a new instance of an app that's not one of the faulty ones -> "Error"
    • Clicking "Backup now" in settings -> "external error"
    $ sudo docker ps
    CONTAINER ID        IMAGE                                                                                COMMAND                  CREATED             STATUS              PORTS                                                          NAMES
    1e837fa522ff        cloudron/img-d45b65559ca80aa3183b07e391f7481d:59530f56-7b28-436f-a769-22795c8570a4   "/app/code/start.sh"     5 hours ago         Up 2 hours          127.0.0.1:45009->3000/tcp                                      43127a5e-8549-4f5b-93c7-d232084786c6
    68898b51f1a8        cloudron/img-062037096d69bbf3ffb5b9316ad89cb9:3630d84e-c9a9-4acf-a148-56c41e2f7de6   "/app/start.sh"          10 hours ago        Up 2 hours          127.0.0.1:45758->80/tcp                                        3e451943-d324-44b4-9839-bc2697da9273
    ff06fdc9d7ec        cloudron/redis:1.0.0                                                                 "/addons/redis/start…"   10 hours ago        Up 2 hours          6379/tcp                                                       redis-3e451943-d324-44b4-9839-bc2697da9273
    d18f9d12c469        cloudron/img-062037096d69bbf3ffb5b9316ad89cb9:ac88ed61-8152-4fa4-b979-c3de05a3bcad   "/home/git/start.sh"     10 hours ago        Up 2 hours          0.0.0.0:22000->22000/tcp, 127.0.0.1:44753->80/tcp              479ba3a4-0884-40a6-8cb3-b22b09ef7e3a
    1321fbd1a042        cloudron/redis:1.0.0                                                                 "/addons/redis/start…"   10 hours ago        Up 2 hours          6379/tcp                                                       redis-479ba3a4-0884-40a6-8cb3-b22b09ef7e3a
    212475e35dca        cloudron/img-d45b65559ca80aa3183b07e391f7481d:c26ed234-86e7-4b55-b44c-083f35458e70   "/app/code/start.sh"     10 hours ago        Up 2 hours          127.0.0.1:45869->3000/tcp                                      96875d98-46cf-475f-a6bb-471467472e03
    61d7e4172fab        cloudron/img-062037096d69bbf3ffb5b9316ad89cb9:f16d9395-c187-4aea-a78f-0b4055f588d8   "/app/code/start.sh"     4 days ago          Up 2 hours          127.0.0.1:40570->3000/tcp                                      6fa5f25c-8930-477f-a81e-d401135ea57e
    201d3a8e3635        cloudron/mail:1.3.0                                                                  "/addons/mail/start.…"   13 days ago         Up 2 hours          127.0.0.1:2020->2020/tcp                                       mail
    b5fb451300b7        cloudron/img-062037096d69bbf3ffb5b9316ad89cb9:32981fcf-f64f-44e4-9a76-0889c47b38d6   "/app/code/start.sh"     2 weeks ago         Up 2 hours          127.0.0.1:41999->8000/tcp                                      7d2e0092-e82f-4061-848a-f291271d08b7
    97a2c51912f4        cloudron/redis:1.0.0                                                                 "/addons/redis/start…"   2 weeks ago         Up 2 hours          6379/tcp                                                       redis-7d2e0092-e82f-4061-848a-f291271d08b7
    b7ff09532027        cloudron/img-d45b65559ca80aa3183b07e391f7481d:d9a912d4-1cb5-42b2-b3c9-c969ae084870   "/app/code/start.sh"     2 weeks ago         Up 2 hours          127.0.0.1:44215->3000/tcp                                      b1e0e80b-e01f-4cce-8c7b-f6cccd71a210
    43b37340b05d        cloudron/redis:1.0.0                                                                 "/addons/redis/start…"   2 weeks ago         Up 2 hours          6379/tcp                                                       redis-b1e0e80b-e01f-4cce-8c7b-f6cccd71a210
    c6aeaf6a1859        cloudron/mongodb:1.1.0                                                               "/addons/mongodb/sta…"   3 weeks ago         Up 2 hours          27017-27018/tcp                                                mongodb
    2fd5adba37e3        cloudron/postgresql:1.1.0                                                            "/addons/postgresql/…"   3 weeks ago         Up 2 hours          5432/tcp                                                       postgresql
    9df63c16a156        cloudron/mysql:1.1.0                                                                 "/addons/mysql/start…"   3 weeks ago         Up 18 minutes       3306/tcp                                                       mysql
    e361c7ebb152        cloudron/graphite:1.0.0                                                              "/app/code/start_gra…"   3 weeks ago         Up 2 hours          127.0.0.1:2003-2004->2003-2004/tcp, 127.0.0.1:8000->8000/tcp   graphite
    


  • @uiharu thanks, if you click on the app icon usually it will display the full error. Does it not say anything?



  • Actually, please try 'sudo systemctl restart docker' and wait for a minute and then reconfigure an app to see if it changes anything.



  • Unfortunately, the "Error fordoamin" overlays are blank (except the buttons)...
    restarting docker...

    service docker status
    ● docker.service - Docker Application Container Engine
       Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
      Drop-In: /etc/systemd/system/docker.service.d
               └─cloudron.conf
       Active: active (running) since Thu 2018-06-14 23:04:06 HKT; 2min 19s ago
         Docs: https://docs.docker.com
     Main PID: 14463 (dockerd)
        Tasks: 283
       Memory: 95.4M
          CPU: 16.229s
       CGroup: /system.slice/docker.service
               ├─14463 /usr/bin/dockerd -H fd:// --log-driver=journald --exec-opt native.cgroupdriver=cgroupfs --storage-driver=overlay2
               ├─14473 docker-containerd --config /var/run/docker/containerd/containerd.toml
               ├─15158 /usr/bin/docker-proxy -proto tcp -host-ip 127.0.0.1 -host-port 8000 -container-ip 172.18.0.2 -container-port 8000
               ├─15169 /usr/bin/docker-proxy -proto tcp -host-ip 127.0.0.1 -host-port 2004 -container-ip 172.18.0.2 -container-port 2004
               ├─15181 /usr/bin/docker-proxy -proto tcp -host-ip 127.0.0.1 -host-port 2003 -container-ip 172.18.0.2 -container-port 2003
               ├─15273 docker-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/e361c7ebb152ab30ae77fae578b7ea1231aecf62bf4335da304523d1f0e8a6
               ├─15654 docker-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/ff06fdc9d7ec7e32cafb23f8ded80ed286a72412ce003cce44ac9e4b59ff63
               ├─15751 docker-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/1321fbd1a0428fe4d2b4913aa223d80ce2272fa1394a2516e4f39c9eae3517
               ├─16013 /usr/bin/docker-proxy -proto tcp -host-ip 127.0.0.1 -host-port 40570 -container-ip 172.18.0.8 -container-port 3000
               ├─16069 docker-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/c6aeaf6a18595164ef0ac459931b63f1a9cde048008045c9f4465ce3cfab3c
               ├─16387 docker-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/61d7e4172fab14ff1404cac998e062e70edb3ffb6aded9e46e9672eefed4d9
               ├─16579 docker-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/9df63c16a156f90faeef606acd76e531314889982497a2079cb62c8c25013f
               ├─16961 docker-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/97a2c51912f48b87ac022abff1c2017370d15b0fe41750896a8aa47fbafc8c
               ├─17150 /usr/bin/docker-proxy -proto tcp -host-ip 127.0.0.1 -host-port 2020 -container-ip 172.18.0.12 -container-port 2020
               ├─17156 docker-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/2fd5adba37e392017558eab064135135aeda30cbfe356f4d4194515edd5b20
               ├─17368 /usr/bin/docker-proxy -proto tcp -host-ip 127.0.0.1 -host-port 44215 -container-ip 172.18.0.6 -container-port 3000
               ├─17431 docker-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/201d3a8e3635cc5bd1faf3339a638f315b07e5710ed53720a3109d7f0f917b
               ├─17689 docker-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/43b37340b05d4b40224c50374c345c31d66f34cbd5a8afadd958c6a4076685
               ├─17758 docker-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/b7ff09532027c3673609e5bd0b551ff21a4ce6a8c7197c09670b5d59e488e7
               ├─18503 /usr/bin/docker-proxy -proto tcp -host-ip 127.0.0.1 -host-port 45009 -container-ip 172.18.0.3 -container-port 3000
               ├─18605 /usr/bin/docker-proxy -proto tcp -host-ip 127.0.0.1 -host-port 45869 -container-ip 172.18.0.16 -container-port 3000
               ├─18684 docker-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/212475e35dca8cee1f601a9b5322b7b9f4e09efbce3fc0c62af489e9c12046
               ├─18743 docker-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/1e837fa522ff01b28b41d01e8e1ae3c0e5c0966f5298d7c35f800f1ab09510
               ├─24207 /usr/bin/docker-proxy -proto tcp -host-ip 127.0.0.1 -host-port 41999 -container-ip 172.18.0.15 -container-port 8000
               ├─24264 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 22000 -container-ip 172.18.0.17 -container-port 22000
               ├─24276 /usr/bin/docker-proxy -proto tcp -host-ip 127.0.0.1 -host-port 44753 -container-ip 172.18.0.17 -container-port 80
               ├─24282 docker-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/b5fb451300b79afaf80549cec6c04371f279b1ce004e0bed950506fff676c8
               ├─24415 docker-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/d18f9d12c4693efe8fb847c10a11a6188b6cb515759bd927fe72193ef3f3c7
               ├─24620 /usr/bin/docker-proxy -proto tcp -host-ip 127.0.0.1 -host-port 45758 -container-ip 172.18.0.14 -container-port 80
               └─24642 docker-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/68898b51f1a87a5075fa30247689c515d6e492adb64f7a7f10e00370da78d0
    
     d18f9d12c469[14463]: [29B blob data]
     d18f9d12c469[14463]: [77B blob data]
     b7ff09532027[14463]: [38B blob data]
     b5fb451300b7[14463]: [83B blob data]
     9df63c16a156[14463]: Waiting for MySQL to come up
     68898b51f1a8[14463]: [99B blob data]
     9df63c16a156[14463]: Waiting for MySQL to come up
     212475e35dca[14463]: [157B blob data]
     c6aeaf6a1859[14463]: 2018-06-14T15:06:25.319+0000 [conn15] update 96875d98-46cf-475f-a6bb-471467472e03.migrations query: { _id: "control" } update: { $set: { version: 12
     c6aeaf6a1859[14463]: 2018-06-14T15:06:25.319+0000 [conn15] command 96875d98-46cf-475f-a6bb-471467472e03.$cmd command: update { update: "migrations", updates: [ { q: { _i
    


  • @girish

    The apps still throw "Error" when i attempt to repair/reconfigure them
    Edit: Uninstalling an app also ends with "Error"



  • @uiharu I sent you a mail from support. I think I will need ssh access to debug what is happening. It's most likely something docker related since the cloudron code itself seems to run fine.



  • Looks like this Cloudron hithttps://forum.cloudron.io/topic/1316/apps-using-mysql-are-not-working/2. MySQL apps are not working. The workaround is also not working.