Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.


    Cloudron Forum

    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular

    Issue with garbage data on Scaleway

    Support
    scaleway backups
    2
    5
    294
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • robi
      robi last edited by girish

      In the event with a problem with a backup, does Cloudron clean up multipart data?

      https://www.scaleway.com/en/docs/s3-multipart-upload/#-Aborting-a-Multipart-Upload

      I just had an issue with the accumulation of all this, causing billing issues and quota issues, which broke their system too 🙂

      Life of Advanced Technology

      girish 1 Reply Last reply Reply Quote 1
      • girish
        girish Staff @robi last edited by

        @robi the cleanup is done automatically by the aws sdk module we use - https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3/ManagedUpload.html ("Handling Multipart Cleanup"). If that's not working with scaleway, might be a good idea to report this issue to them.

        robi 1 Reply Last reply Reply Quote 0
        • robi
          robi @girish last edited by

          @girish when backups fail and retry, it seems to cause this as evidenced by multiple "directories" in the object store of the SAME date and different session strings.

          So I wouldn't know if this is an SDK problem or an implementation of the SDK.

          I am attempting to clean up 10s of GB of these extra directories, but their system is not so great at deletions (plagued with timeouts and their poor browser based client that sends chatty messages back and forth for every object).

          Life of Advanced Technology

          girish 1 Reply Last reply Reply Quote 0
          • girish
            girish Staff @robi last edited by

            @robi Oh, seeing session strings in the UI seems definitely like some backend issue. I am not sure what the client can do about those. Did their support say anything about this?

            robi 1 Reply Last reply Reply Quote 0
            • robi
              robi @girish last edited by

              @girish no, they're pretty useless. Their web UI S3 console is such crap it can't handle the chatty API requests and keeps timing out. Also I may be wrong that multiple directories are because of failures and restarts. It just looks like multiple changed apps per day get a new dir.

              So I am attempting other workarounds. Like creating a new bucket and just nuking the old one.

              rsync isn't great for object store backups as it makes a ton of small files.
              tgz isn't great as it's a lot of repeated information.

              We need something hybrid that is the best of both.
              Something like backing up to a local Minio much more quickly then doing an object to object store transfer offsite, which is much more efficient. This may also offer an opportunity to dedupe and further optimize.

              Life of Advanced Technology

              1 Reply Last reply Reply Quote 0
              • First post
                Last post
              Powered by NodeBB