Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.


Skip to content
  • Categories
  • Recent
  • Tags
  • Popular
  • Bookmarks
  • Search
Skins
  • Light
  • Cerulean
  • Cosmo
  • Flatly
  • Journal
  • Litera
  • Lumen
  • Lux
  • Materia
  • Minty
  • Morph
  • Pulse
  • Sandstone
  • Simplex
  • Sketchy
  • Spacelab
  • United
  • Yeti
  • Zephyr
  • Dark
  • Cyborg
  • Darkly
  • Quartz
  • Slate
  • Solar
  • Superhero
  • Vapor

  • Default (No Skin)
  • No Skin
Collapse
Brand Logo

Cloudron Forum

Apps | Demo | Docs | Install
  1. Cloudron Forum
  2. Support
  3. Backup timeout

Backup timeout

Scheduled Pinned Locked Moved Solved Support
backupspostgresql
10 Posts 2 Posters 883 Views 2 Watching
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • A Offline
    A Offline
    arnaudguy
    wrote on last edited by joseph
    #1

    Hello,

    Every day I encounter an issue where Cloudron restarts in the middle of the backup process. This causes the backups to fail. Is it possible to trigger backups via the command line?

    Attached are the logs. Thank you in advance for your insights and explanations.

    Dec 04 11:58:10 box:apphealthmonitor app health: 20 running / 0 stopped / 0 unresponsive
    Dec 04 11:58:30 box:apphealthmonitor setHealth: ece432d9-f98a-4ab4-ad09-bafa1cdaff08 (desk.xx.xx) waiting for 1179.554 to update health
    Dec 04 11:58:55 box:locker Lock unreleased full_backup
    Dec 04 11:58:55 ServerError [ServiceUnavailableError]: Response timeout
    Dec 04 11:58:55 at IncomingMessage.<anonymous> (/home/yellowtent/box/node_modules/connect-timeout/index.js:84:8)
    Dec 04 11:58:55 at IncomingMessage.emit (node:events:519:28)
    Dec 04 11:58:55 at IncomingMessage.emit (node:domain:488:12)
    Dec 04 11:58:55 at Timeout._onTimeout (/home/yellowtent/box/node_modules/connect-timeout/index.js:49:11)
    Dec 04 11:58:55 at listOnTimeout (node:internal/timers:581:17)
    Dec 04 11:58:55 at process.processTimers (node:internal/timers:519:7) {
    Dec 04 11:58:55 code: 'ETIMEDOUT',
    Dec 04 11:58:55 timeout: 60000
    Dec 04 11:58:55 }
    Dec 04 11:58:55 ServerError [ServiceUnavailableError]: Response timeout
    Dec 04 11:58:55 at IncomingMessage.<anonymous> (/home/yellowtent/box/node_modules/connect-timeout/index.js:84:8)
    Dec 04 11:58:55 at IncomingMessage.emit (node:events:519:28)
    Dec 04 11:58:55 at IncomingMessage.emit (node:domain:488:12)
    Dec 04 11:58:55 at Timeout._onTimeout (/home/yellowtent/box/node_modules/connect-timeout/index.js:49:11)
    Dec 04 11:58:55 at listOnTimeout (node:internal/timers:581:17)
    Dec 04 11:58:55 at process.processTimers (node:internal/timers:519:7) {
    Dec 04 11:58:55 code: 'ETIMEDOUT',
    Dec 04 11:58:55 timeout: 60000
    Dec 04 11:58:55 }
    Dec 04 11:59:20 AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:
    Dec 04 11:59:20 + actual - expected
    Dec 04 11:59:20 2024-12-04T10:59:20.702Z
    Dec 04 11:59:20 + 'undefined'
    Dec 04 11:59:20 - 'string'
    Dec 04 11:59:20 at get (/home/yellowtent/box/src/routes/tasks.js:33:12)
    Dec 04 11:59:20 at Layer.handle [as handle_request] (/home/yellowtent/box/node_modules/express/lib/router/layer.js:95:5)
    Dec 04 11:59:20 at next (/home/yellowtent/box/node_modules/express/lib/router/route.js:149:13)
    Dec 04 11:59:20 at load (/home/yellowtent/box/src/routes/tasks.js:29:5)
    Dec 04 11:59:20 at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    Dec 04 11:59:22 box:server ==========================================
    Dec 04 11:59:22 box:server Cloudron 8.1.0
    Dec 04 11:59:22 box:server ==========================================
    Dec 04 11:59:22 box:platform initialize: start platform
    Dec 04 11:59:22 box:tasks stopAllTasks: stopping all tasks
    Dec 04 11:59:22 box:shell tasks /usr/bin/sudo -S /home/yellowtent/box/src/scripts/stoptask.sh all
    Dec 04 11:59:22 box:shell box-task-16616.service loaded failed failed /home/yellowtent/box/src/scripts/../taskworker.js 16616 /home/yellowtent/platformdata/logs/tasks/16616.log
    Dec 04 11:59:22 box-task-16617.service loaded failed failed /home/yellowtent/box/src/scripts/../taskworker.js 16617 /home/yellowtent/platformdata/logs/tasks/16617.log
    Dec 04 11:59:22
    Dec 04 11:59:22 box:shell All tasks stopped
    
    
    2024-12-04T10:57:52.611Z box:tasks update 16618: {"percent":18.391304347826086,"message":"Backing up matrix.xx.xx (5/20)"}
    2024-12-04T10:57:52.613Z box:tasks update 16618: {"percent":22.73913043478261,"message":"Snapshotting app matrix.xx.xx"}
    2024-12-04T10:57:52.614Z box:services backupAddons
    2024-12-04T10:57:52.614Z box:services backupAddons: backing up ["localstorage","oidc","postgresql","sendmail","turn"]
    2024-12-04T10:57:52.614Z box:services Backing up postgresql
    2024-12-04T10:57:53.188Z box:services pipeRequestToFile: connected with status code 200
    2024-12-04T10:59:07.681Z box:backuptask snapshotApp: matrix.xx.xx took 75.068 seconds
    2024-12-04T10:59:07.681Z box:tasks update 16618: {"percent":22.73913043478261,"message":"Uploading app snapshot matrix.xx.xx"}
    2024-12-04T10:59:07.682Z box:backuptask runBackupUpload: adjusting heap size to 8192M
    2024-12-04T10:59:07.682Z box:shell backuptask /usr/bin/sudo -S -E --close-from=4 /home/yellowtent/box/src/scripts/backupupload.js snapshot/app_51b362b8-a595-476a-8376-cda3beb8a24c tgz {"localRoot":"/home/yellowtent/appsdata/51b362b8-a595-476a-8376-cda3beb8a24c","layout":[]}
    2024-12-04T10:59:12.046Z box:backupupload Backing up {"localRoot":"/home/yellowtent/appsdata/51b362b8-a595-476a-8376-cda3beb8a24c","layout":[]} to snapshot/app_51b362b8-a595-476a-8376-cda3beb8a24c
    2024-12-04T10:59:12.067Z box:backuptask upload: path snapshot/app_51b362b8-a595-476a-8376-cda3beb8a24c format tgz dataLayout {"localRoot":"/home/yellowtent/appsdata/51b362b8-a595-476a-8376-cda3beb8a24c","layout":[]}
    2024-12-04T10:59:12.078Z box:backuptask checkPreconditions: mount point status is {"state":"active"}&&&&&
    2024-12-04T10:59:12.182Z box:backuptask checkPreconditions: getting disk usage of /home/yellowtent/appsdata/51b362b8-a595-476a-8376-cda3beb8a24c
    2024-12-04T10:59:12.182Z box:shell backuptask: du --dereference-args --summarize --block-size=1 --exclude=*.lock --exclude=dovecot.list.index.log.* /home/yellowtent/appsdata/51b362b8-a595-476a-8376-cda3beb8a24c
    
    1 Reply Last reply
    1
    • nebulonN Offline
      nebulonN Offline
      nebulon
      Staff
      wrote on last edited by nebulon
      #2

      That is a very strange crash at https://git.cloudron.io/platform/box/-/blob/master/src/routes/tasks.js?ref_type=heads#L33 which is only every called via https://git.cloudron.io/platform/box/-/blob/master/src/server.js?ref_type=heads#L141

      Can't see how that route handler would be called unless the code is somehow changed. But it really looks like some execution error caused by that intermediate timeout.

      1 Reply Last reply
      0
      • nebulonN nebulon marked this topic as a question on
      • A Offline
        A Offline
        arnaudguy
        wrote on last edited by
        #3

        Hi, I've see it's always crash during the snapshot (dump postgresql) of gitlab.

        1 Reply Last reply
        0
        • nebulonN Offline
          nebulonN Offline
          nebulon
          Staff
          wrote on last edited by
          #4

          Do you see any further errors in the postgresql service logs during such times? Maybe that runs out of memory for example?

          1 Reply Last reply
          0
          • A Offline
            A Offline
            arnaudguy
            wrote on last edited by
            #5

            no i don't see that. But I have a server that is very loaded with processes.

            1 Reply Last reply
            0
            • A Offline
              A Offline
              arnaudguy
              wrote on last edited by
              #6

              Capture d’écran 2024-12-04 à 21.38.58.png

              1 Reply Last reply
              0
              • A Offline
                A Offline
                arnaudguy
                wrote on last edited by
                #7

                Hi,
                Yesterday night I'have the same situation. Now i'm sure it's was alaways on backup of gitlab.

                Thanks
                Capture d’écran 2024-12-05 à 07.22.49.png Capture d’écran 2024-12-05 à 07.21.53.png

                1 Reply Last reply
                0
                • nebulonN Offline
                  nebulonN Offline
                  nebulon
                  Staff
                  wrote on last edited by
                  #8

                  We may have to debug this on your instance then directly. Can you send us a mail to support@cloudron.io and enable remote SSH support for us? https://docs.cloudron.io/support/#remote-support

                  1 Reply Last reply
                  1
                  • A Offline
                    A Offline
                    arnaudguy
                    wrote on last edited by
                    #9

                    Hi, Thank you, I've send email.

                    1 Reply Last reply
                    0
                    • A Offline
                      A Offline
                      arnaudguy
                      wrote on last edited by
                      #10

                      The issue is now resolved. The problem was with the GitLab database, where we found 160 GB of logs. The database dump was failing because there wasn’t enough free disk space. We have purged the database, and backups are now successful.

                      Thank you for your support.

                      1 Reply Last reply
                      1
                      • nebulonN nebulon has marked this topic as solved on
                      Reply
                      • Reply as topic
                      Log in to reply
                      • Oldest to Newest
                      • Newest to Oldest
                      • Most Votes


                      • Login

                      • Don't have an account? Register

                      • Login or register to search.
                      • First post
                        Last post
                      0
                      • Categories
                      • Recent
                      • Tags
                      • Popular
                      • Bookmarks
                      • Search