Backup import fails on new Cloudron instance

[yusf]

When restoring Gitea app data from an DO S3 object storage to a new Gitea app on a new Cloudron instance, I get this error:

Task Error: Task 3 crashed with code 255 and signal null

The log states that this happens during status: installationState": "pending_restore" which is obvious when watching the process.

I need to migrate this app over to this new instance. Where to start resolving this?

[girish]

@yusf Can you paste the app logs (the final few lines) ?

[yusf]

@girish

Jan 28 11:00:43 2021-01-28 10:00:43,399 WARN received SIGTERM indicating exit request
Jan 28 11:00:43 2021-01-28 10:00:43,403 INFO waiting for sshd, gitea to die
Jan 28 11:00:43 2021/01/28 10:00:43 ...eful/manager_unix.go:127:handleSignals() [W] PID 56. Received SIGTERM. Shutting down...
Jan 28 11:00:43 2021/01/28 10:00:43 cmd/web.go:208:runWeb() [I] HTTP Listener: 0.0.0.0:3000 Closed
Jan 28 11:00:43 2021/01/28 10:00:43 ...eful/server_hooks.go:47:doShutdown() [I] PID: 56 Listener ([::]:3000) closed.
Jan 28 11:00:43 2021/01/28 10:00:43 .../graceful/manager.go:217:doHammerTime() [W] Setting Hammer condition
Jan 28 11:00:44 2021/01/28 10:00:44 .../graceful/manager.go:231:doTerminate() [W] Terminating
Jan 28 11:00:44 2021/01/28 10:00:44 ...er/issues/indexer.go:169:2() [I] PID: 56 Issue Indexer closed
Jan 28 11:00:44 2021/01/28 10:00:44 cmd/web.go:210:runWeb() [I] PID: 56 Gitea Web Finished
Jan 28 11:00:44 2021-01-28 10:00:44,484 INFO stopped: gitea (exit status 0)
Jan 28 11:00:44 2021-01-28 10:00:44,516 INFO stopped: sshd (exit status 0)

[nebulon]

@yusf Can you maybe look into the full logs to find more information, what you posted looks like a shutdown of the app only.

[yusf]

@nebulon Right, so this is what's seen in the app log when restoring. Cleared the view first to make sure to capture the relevant parts.

Jan 31 13:51:47 box:settings initCache: pre-load settings
Jan 31 13:51:48 box:taskworker Starting task 28. Logs are at /home/yellowtent/platformdata/logs/APP_ID/apptask.log
Jan 31 13:51:48 box:tasks 28: {"percent":2,"error":null}
Jan 31 13:51:48 box:apptask gitea.domain.tld startTask installationState: pending_restore runState: running
Jan 31 13:51:48 box:tasks 28: {"percent":10,"message":"Cleaning up old install"}
Jan 31 13:51:48 box:shell reload spawn: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/reloadnginx.sh
Jan 31 13:51:48 box:apptask gitea.domain.tld deleting app containers (app, scheduler)
Jan 31 13:51:48 box:shell removeCollectdProfile spawn: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/configurecollectd.sh remove APP_ID
Jan 31 13:51:48 box:shell removeCollectdProfile (stdout): Restarting collectd
Jan 31 13:51:48 box:shell removeCollectdProfile (stdout): Removing collectd stats of APP_ID
Jan 31 13:51:48 box:shell removeLogrotateConfig spawn: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/configurelogrotate.sh remove APP_ID
Jan 31 13:51:48 box:apptask gitea.domain.tld updating app with values: {"containerId":null}
Jan 31 13:51:48 box:addons gitea.domain.tld teardownAddons: Tearing down []
Jan 31 13:51:48 box:apptask gitea.domain.tld updating app with values: {"containerIp":"172.18.18.225"}
Jan 31 13:51:48 box:tasks 28: {"percent":20,"message":"Downloading icon"}
Jan 31 13:51:48 box:apptask gitea.domain.tld Downloading icon of io.gitea.cloudronapp@1.19.1
Jan 31 13:51:49 box:tasks 28: {"percent":30,"message":"Registering subdomains"}
Jan 31 13:51:49 box:apptask gitea.domain.tld registerSubdomain: Will register [{"subdomain":"gitea","domain":"domain.tld"}]
Jan 31 13:51:49 box:apptask gitea.domain.tld Registering subdomain: gitea.domain.tld
Jan 31 13:51:49 box:domains upsertDNSRecord: gitea on domain.tld type A values [ 'IP_ADRESS' ]
Jan 31 13:51:49 box:dns/manual upsert: gitea for zone domain.tld of type A with values ["IP_ADRESS"]
Jan 31 13:51:49 box:tasks 28: {"percent":40,"message":"Downloading image"}
Jan 31 13:51:49 box:docker downloadImage cloudron/io.gitea.cloudronapp:20201228-202120-569e64a22
Jan 31 13:51:49 box:tasks 28: {"percent":50,"message":"Creating app data directory"}
Jan 31 13:51:49 box:tasks 28: {"percent":65,"message":"Download backup and restoring addons"}
Jan 31 13:51:49 box:addons gitea.domain.tld setupAddons: Setting up ["mysql","sendmail","localstorage","ldap"]
Jan 31 13:51:49 box:addons gitea.domain.tld Setting up addon mysql with options {}
Jan 31 13:51:49 box:addons gitea.domain.tld Setting up mysql
Jan 31 13:51:49 box:addons gitea.domain.tld Setting mysql addon config to [{"name":"CLOUDRON_MYSQL_USERNAME","value":"USER"},{"name":"CLOUDRON_MYSQL_PASSWORD","value":"PASS"},{"name":"CLOUDRON_MYSQL_HOST","value":"mysql"},{"name":"CLOUDRON_MYSQL_PORT","value":"3306"},{"name":"CLOUDRON_MYSQL_URL","value":"mysql://USER:PASS@mysql/USER"},{"name":"CLOUDRON_MYSQL_DATABASE","value":"USER"}]
Jan 31 13:51:49 box:addons gitea.domain.tld Setting up addon sendmail with options {}
Jan 31 13:51:49 box:addons gitea.domain.tld Setting up SendMail
Jan 31 13:51:49 box:addons gitea.domain.tld Setting sendmail addon config to [{"name":"CLOUDRON_MAIL_SMTP_SERVER","value":"mail"},{"name":"CLOUDRON_MAIL_SMTP_PORT","value":"2525"},{"name":"CLOUDRON_MAIL_SMTPS_PORT","value":"2465"},{"name":"CLOUDRON_MAIL_SMTP_USERNAME","value":"gitea.app@domain.tld"},{"name":"CLOUDRON_MAIL_SMTP_PASSWORD","value":"PASS_SMTP"},{"name":"CLOUDRON_MAIL_FROM","value":"gitea.app@domain.tld"},{"name":"CLOUDRON_MAIL_DOMAIN","value":"domain.tld"}]
Jan 31 13:51:49 box:addons gitea.domain.tld Setting up addon localstorage with options {}
Jan 31 13:51:49 box:addons gitea.domain.tld setupLocalStorage
Jan 31 13:51:49 box:shell createVolume spawn: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/mkdirvolume.sh /home/yellowtent/appsdata/APP_ID/data
Jan 31 13:51:49 box:addons gitea.domain.tld Setting up addon ldap with options {}
Jan 31 13:51:49 box:addons gitea.domain.tld Setting up LDAP
Jan 31 13:51:49 box:addons gitea.domain.tld clearAddons
Jan 31 13:51:49 box:addons gitea.domain.tld clearAddons: clearing ["mysql","sendmail","localstorage","ldap"]
Jan 31 13:51:49 box:addons gitea.domain.tld clearLocalStorage
Jan 31 13:51:49 box:shell clearVolume spawn: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/clearvolume.sh clear /home/yellowtent/appsdata/APP_ID/data
Jan 31 13:51:49 box:backups download: Downloading 2021-01-30-220006-927/app_FORMER_APP_ID_2021-01-30-221251-850_v1.19.1 of format tgz to {"localRoot":"/home/yellowtent/appsdata/APP_ID","layout":[]}

At this point it's not getting any further, while at the same time it throws Error : Task Error - Task 28 crashed with code 255 and signal null in the UI.

Box log shows:

{
  "app": {
    "id": "APP_ID",
    "appStoreId": "io.gitea.cloudronapp",
    "installationState": "error",
    "runState": "running",
    "health": "healthy",
    "containerId": null,
    "memoryLimit": 0,
    "cpuShares": 512,
    "label": null,
    "taskId": null,
    "sso": true,
    "enableBackup": true,
    "proxyAuth": false,
    "containerIp": "172.18.19.60",
    "creationTime": "2021-01-28T09:19:42.000Z",
    "updateTime": "2021-01-28T09:19:42.000Z",
    "mailboxName": "gitea.app",
    "mailboxDomain": "domain.tld",
    "enableAutomaticUpdate": true,
    "dataDir": null,
    "ts": "2021-01-31T12:51:11.000Z",
    "healthTime": "2021-01-28T10:00:40.000Z",
    "location": "gitea",
    "domain": "domain.tld",
    "alternateDomains": [],
    "manifest": {
      "id": "io.gitea.cloudronapp",
      "title": "Gitea",
      "author": "Gitea developers",
      "description": "This app packages Gitea <upstream>1.13.0</upstream>\n\nGitea is a painless self-hosted Git service. It is similar to GitHub, Bitbucket or Gitlab.\n\n### Purpose\n\nThe goal of this project is to make the easiest, fastest, and most painless way to set up a self-hosted Git service.\n\n### Features\n\n- Activity timeline\n- SSH and HTTP/HTTPS protocols\n- SMTP/LDAP/Reverse proxy authentication\n- Reverse proxy with sub-path\n- Account/Organization/Repository management\n- Repository/Organization webhooks (including Slack)\n- Repository Git hooks/deploy keys\n- Repository issues, pull requests and wiki\n- Add/Remove repository collaborators\n- Gravatar and custom source\n- Mail service\n- Administration panel\n\n### Bug reports\n\nOpen bugs on [Github](https://git.cloudron.io/cloudron/gitea-app/issues)\n",
      "tagline": "A painless self-hosted Git Service",
      "version": "1.19.1",
      "healthCheckPath": "/healthcheck",
      "httpPort": 3000,
      "memoryLimit": 536870912,
      "addons": {
        "mysql": {},
        "sendmail": {},
        "localstorage": {},
        "ldap": {}
      },
      "tcpPorts": {
        "SSH_PORT": {
          "title": "SSH Port",
          "description": "SSH Port over which repos can be pushed & pulled",
          "defaultValue": 29418
        }
      },
      "manifestVersion": 2,
      "website": "https://gitea.io",
      "contactEmail": "apps@cloudron.io",
      "icon": "file://logo.png",
      "optionalSso": true,
      "mediaLinks": [
        "https://screenshots.cloudron.io/io.gitea.cloudronapp/1.png",
        "https://screenshots.cloudron.io/io.gitea.cloudronapp/2.png",
        "https://screenshots.cloudron.io/io.gitea.cloudronapp/3.png",
        "https://screenshots.cloudron.io/io.gitea.cloudronapp/4.png",
        "https://screenshots.cloudron.io/io.gitea.cloudronapp/5.png"
      ],
      "tags": [
        "version control",
        "git",
        "code hosting",
        "development",
        "github",
        "bitbucket",
        "gitlab"
      ],
      "changelog": "* Update Gitea to 1.13.1\n* [Full changelog](https://github.com/go-gitea/gitea/releases/tag/v1.13.1)\n* Security: Hide private participation in Orgs (#13994) (#14031)\n* Security: Fix escaping issue in diff (#14153) (#14154)\n",
      "postInstallMessage": "This app is pre-setup with an admin account (use the `Local` authentication source for logging in as admin).\nThe initial credentials are:\n\n**Username**: root<br/>\n**Password**: changeme<br/>\n\nPlease change the admin password immediately.\n\n<sso>\nCloudron users can login using the `Cloudron` authentication source.\n</sso>\n\n",
      "minBoxVersion": "5.3.0",
      "forumUrl": "https://forum.cloudron.io/category/19/gitea",
      "documentationUrl": "https://docs.cloudron.io/apps/gitea/",
      "dockerImage": "cloudron/io.gitea.cloudronapp:20201228-202120-569e64a22"
    },
    "tags": [],
    "reverseProxyConfig": {},
    "portBindings": {
      "SSH_PORT": 2222
    },
    "accessRestriction": null,
    "debugMode": null,
    "servicesConfig": {},
    "env": {},
    "mounts": [],
    "error": {
      "message": "Task 27 crashed with code 255 and signal null",
      "reason": "Task Error",
      "crashed": true,
      "stopped": false,
      "taskId": "27",
      "installationState": "pending_restore"
    },
    "iconUrl": "/api/v1/apps/APP_ID/icon",
    "fqdn": "gitea.domain.tld"
  },
  "backupId": "2021-01-30-220006-927/app_FORMER_APP_ID_2021-01-30-221251-850_v1.19.1",
  "fromManifest": {
    "id": "io.gitea.cloudronapp",
    "title": "Gitea",
    "author": "Gitea developers",
    "description": "This app packages Gitea <upstream>1.13.0</upstream>\n\nGitea is a painless self-hosted Git service. It is similar to GitHub, Bitbucket or Gitlab.\n\n### Purpose\n\nThe goal of this project is to make the easiest, fastest, and most painless way to set up a self-hosted Git service.\n\n### Features\n\n- Activity timeline\n- SSH and HTTP/HTTPS protocols\n- SMTP/LDAP/Reverse proxy authentication\n- Reverse proxy with sub-path\n- Account/Organization/Repository management\n- Repository/Organization webhooks (including Slack)\n- Repository Git hooks/deploy keys\n- Repository issues, pull requests and wiki\n- Add/Remove repository collaborators\n- Gravatar and custom source\n- Mail service\n- Administration panel\n\n### Bug reports\n\nOpen bugs on [Github](https://git.cloudron.io/cloudron/gitea-app/issues)\n",
    "tagline": "A painless self-hosted Git Service",
    "version": "1.19.1",
    "healthCheckPath": "/healthcheck",
    "httpPort": 3000,
    "memoryLimit": 536870912,
    "addons": {
      "mysql": {},
      "sendmail": {},
      "localstorage": {},
      "ldap": {}
    },
    "tcpPorts": {
      "SSH_PORT": {
        "title": "SSH Port",
        "description": "SSH Port over which repos can be pushed & pulled",
        "defaultValue": 29418
      }
    },
    "manifestVersion": 2,
    "website": "https://gitea.io",
    "contactEmail": "apps@cloudron.io",
    "icon": "file://logo.png",
    "optionalSso": true,
    "mediaLinks": [
      "https://screenshots.cloudron.io/io.gitea.cloudronapp/1.png",
      "https://screenshots.cloudron.io/io.gitea.cloudronapp/2.png",
      "https://screenshots.cloudron.io/io.gitea.cloudronapp/3.png",
      "https://screenshots.cloudron.io/io.gitea.cloudronapp/4.png",
      "https://screenshots.cloudron.io/io.gitea.cloudronapp/5.png"
    ],
    "tags": [
      "version control",
      "git",
      "code hosting",
      "development",
      "github",
      "bitbucket",
      "gitlab"
    ],
    "changelog": "* Update Gitea to 1.13.1\n* [Full changelog](https://github.com/go-gitea/gitea/releases/tag/v1.13.1)\n* Security: Hide private participation in Orgs (#13994) (#14031)\n* Security: Fix escaping issue in diff (#14153) (#14154)\n",
    "postInstallMessage": "This app is pre-setup with an admin account (use the `Local` authentication source for logging in as admin).\nThe initial credentials are:\n\n**Username**: root<br/>\n**Password**: changeme<br/>\n\nPlease change the admin password immediately.\n\n<sso>\nCloudron users can login using the `Cloudron` authentication source.\n</sso>\n\n",
    "minBoxVersion": "5.3.0",
    "forumUrl": "https://forum.cloudron.io/category/19/gitea",
    "documentationUrl": "https://docs.cloudron.io/apps/gitea/",
    "dockerImage": "cloudron/io.gitea.cloudronapp:20201228-202120-569e64a22"
  },
  "toManifest": {
    "id": "io.gitea.cloudronapp",
    "title": "Gitea",
    "author": "Gitea developers",
    "description": "This app packages Gitea <upstream>1.13.0</upstream>\n\nGitea is a painless self-hosted Git service. It is similar to GitHub, Bitbucket or Gitlab.\n\n### Purpose\n\nThe goal of this project is to make the easiest, fastest, and most painless way to set up a self-hosted Git service.\n\n### Features\n\n- Activity timeline\n- SSH and HTTP/HTTPS protocols\n- SMTP/LDAP/Reverse proxy authentication\n- Reverse proxy with sub-path\n- Account/Organization/Repository management\n- Repository/Organization webhooks (including Slack)\n- Repository Git hooks/deploy keys\n- Repository issues, pull requests and wiki\n- Add/Remove repository collaborators\n- Gravatar and custom source\n- Mail service\n- Administration panel\n\n### Bug reports\n\nOpen bugs on [Github](https://git.cloudron.io/cloudron/gitea-app/issues)\n",
    "tagline": "A painless self-hosted Git Service",
    "version": "1.19.1",
    "healthCheckPath": "/healthcheck",
    "httpPort": 3000,
    "memoryLimit": 536870912,
    "addons": {
      "mysql": {},
      "sendmail": {},
      "localstorage": {},
      "ldap": {}
    },
    "tcpPorts": {
      "SSH_PORT": {
        "title": "SSH Port",
        "description": "SSH Port over which repos can be pushed & pulled",
        "defaultValue": 29418
      }
    },
    "manifestVersion": 2,
    "website": "https://gitea.io",
    "contactEmail": "apps@cloudron.io",
    "icon": "file://logo.png",
    "optionalSso": true,
    "mediaLinks": [
      "https://screenshots.cloudron.io/io.gitea.cloudronapp/1.png",
      "https://screenshots.cloudron.io/io.gitea.cloudronapp/2.png",
      "https://screenshots.cloudron.io/io.gitea.cloudronapp/3.png",
      "https://screenshots.cloudron.io/io.gitea.cloudronapp/4.png",
      "https://screenshots.cloudron.io/io.gitea.cloudronapp/5.png"
    ],
    "tags": [
      "version control",
      "git",
      "code hosting",
      "development",
      "github",
      "bitbucket",
      "gitlab"
    ],
    "changelog": "* Update Gitea to 1.13.1\n* [Full changelog](https://github.com/go-gitea/gitea/releases/tag/v1.13.1)\n* Security: Hide private participation in Orgs (#13994) (#14031)\n* Security: Fix escaping issue in diff (#14153) (#14154)\n",
    "postInstallMessage": "This app is pre-setup with an admin account (use the `Local` authentication source for logging in as admin).\nThe initial credentials are:\n\n**Username**: root<br/>\n**Password**: changeme<br/>\n\nPlease change the admin password immediately.\n\n<sso>\nCloudron users can login using the `Cloudron` authentication source.\n</sso>\n\n",
    "minBoxVersion": "5.3.0",
    "forumUrl": "https://forum.cloudron.io/category/19/gitea",
    "documentationUrl": "https://docs.cloudron.io/apps/gitea/",
    "dockerImage": "cloudron/io.gitea.cloudronapp:20201228-202120-569e64a22"
  },
  "taskId": "28"
}

[girish]

@yusf Are on Cloudron 6.1 ? Also, can you check dmesg output to see if there are any oom errors?

[yusf]

@girish Both Cloudrons are on 6.0.1. I'm not sure how to use dmesg. dmesg | grep oom doesn't yield any results at least.

[yusf]

@yusf From /home/yellowtent/platformdata/logs/box.log but nothing useful from what I can tell.

2021-02-01T09:52:43.771Z box:shell startTask code: 255, signal: null
2021-02-01T09:52:43.774Z box:tasks startTask: 40 completed with code 255 and signal null
2021-02-01T09:52:43.780Z box:tasks setCompleted - 40: {"error":{"message":"Task 40 crashed with code 255 and signal null","code":"crashed"}}
2021-02-01T09:52:43.781Z box:tasks 40: {"percent":100,"error":{"message":"Task 40 crashed with code 255 and signal null","code":"crashed"}}
2021-02-01T09:52:43.781Z box:apps scheduleTask: task 40 of NEW_APP_ID completed
2021-02-01T09:52:43.781Z box:apps Apptask crashed/stopped: Task 40 crashed with code 255 and signal null
2021-02-01T09:52:43.783Z box:locker Released : apptask
2021-02-01T09:52:43.783Z box:sftp rebuilding container
2021-02-01T09:52:43.787Z box:scheduler resumeJobs: NEW_APP_ID
2021-02-01T09:52:43.788Z box:tasks startTask: 40 done
2021-02-01T09:52:43.802Z box:shell inspectSftp exec: docker inspect --format="{{json .Mounts }}" sftp
2021-02-01T09:52:44.211Z box:shell inspectSftp (stdout): [{"Type":"bind","Source":"/home/yellowtent/boxdata/sftp/ssh","Destination":"/etc/ssh","Mode":"ro","RW":false,"Propagation":"rprivate"},{"Type":"volume","Name":"SOURCE_VOL","Source":"/var/lib/docker/volumes/SOURCE_VOL/_data","Destination":"/run","Driver":"local","Mode":"","RW":true,"Propagation":""},{"Type":"volume","Name":"DEST_VOL","Source":"/var/lib/docker/volumes/DEST_VOL/_data","Destination":"/tmp","Driver":"local","Mode":"","RW":true,"Propagation":""},{"Type":"bind","Source":"/home/yellowtent/appsdata/NEW_APP_ID/data","Destination":"/app/data/NEW_APP_ID","Mode":"","RW":true,"Propagation":"rprivate"}]

2021-02-01T09:52:44.211Z box:shell inspectSftp (stderr): 
2021-02-01T09:52:44.213Z box:sftp Skipping rebuild, no changes
2021-02-01T09:52:50.006Z box:apphealthmonitor app health: 0 alive / 1 dead.

[nebulon]

@yusf can you also check the logs of the failing task? Those should be located at /home/yellowtent/platformdata/logs/tasks/40.log

[yusf]

@nebulon Alas there is no 40.log. I ran the restoration task again and the new task 41 didn't show up as 41.log either.

[girish]

@yusf Strange. Do you have enough disk space? Can you give the server a reboot and see if the logs files are getting generated?

[yusf]

@girish I have plenty of disk space, 14 GB. I'll try rebooting first.

[yusf]

No difference in rebooting. This is getting frustrating. Are there other, perhaps manual ways of restoring? For instance I can migrate the app backup to the new server manually if I can import it to the new app somehow.

[girish]

@yusf If you can contact us on support@cloudron.io, we can try to figure out what is happening.

[yusf]

I'm pressed for time to follow up currently, but I managed to solve perform the migration by

1. Configuring backup target back to the local /var/backups
2. Performing a fresh app backup on the old server
3. Add SSH key of new server to old server's authorized_keys
4. Using scp to transfer the app backup from old server to new.
5. Replace the latest local Gitea backup on the new server with the transferred one.
6. Restore to the latest app backup on the new server.

[girish]

@yusf OK, thanks, when you find time, feel free to reach on the support mailbox.