Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.


Skip to content
  • Categories
  • Recent
  • Tags
  • Popular
  • Bookmarks
  • Search
Skins
  • Light
  • Cerulean
  • Cosmo
  • Flatly
  • Journal
  • Litera
  • Lumen
  • Lux
  • Materia
  • Minty
  • Morph
  • Pulse
  • Sandstone
  • Simplex
  • Sketchy
  • Spacelab
  • United
  • Yeti
  • Zephyr
  • Dark
  • Cyborg
  • Darkly
  • Quartz
  • Slate
  • Solar
  • Superhero
  • Vapor

  • Default (No Skin)
  • No Skin
Collapse
Brand Logo

Cloudron Forum

Apps | Demo | Docs | Install
  1. Cloudron Forum
  2. Support
  3. Issue with garbage data on Scaleway

Issue with garbage data on Scaleway

Scheduled Pinned Locked Moved Support
scalewaybackups
5 Posts 2 Posters 1.1k Views 2 Watching
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • robiR Offline
    robiR Offline
    robi
    wrote on last edited by girish
    #1

    In the event with a problem with a backup, does Cloudron clean up multipart data?

    https://www.scaleway.com/en/docs/s3-multipart-upload/#-Aborting-a-Multipart-Upload

    I just had an issue with the accumulation of all this, causing billing issues and quota issues, which broke their system too 🙂

    Conscious tech

    girishG 1 Reply Last reply
    1
    • robiR robi

      In the event with a problem with a backup, does Cloudron clean up multipart data?

      https://www.scaleway.com/en/docs/s3-multipart-upload/#-Aborting-a-Multipart-Upload

      I just had an issue with the accumulation of all this, causing billing issues and quota issues, which broke their system too 🙂

      girishG Do not disturb
      girishG Do not disturb
      girish
      Staff
      wrote on last edited by
      #2

      @robi the cleanup is done automatically by the aws sdk module we use - https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3/ManagedUpload.html ("Handling Multipart Cleanup"). If that's not working with scaleway, might be a good idea to report this issue to them.

      robiR 1 Reply Last reply
      0
      • girishG girish

        @robi the cleanup is done automatically by the aws sdk module we use - https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3/ManagedUpload.html ("Handling Multipart Cleanup"). If that's not working with scaleway, might be a good idea to report this issue to them.

        robiR Offline
        robiR Offline
        robi
        wrote on last edited by
        #3

        @girish when backups fail and retry, it seems to cause this as evidenced by multiple "directories" in the object store of the SAME date and different session strings.

        So I wouldn't know if this is an SDK problem or an implementation of the SDK.

        I am attempting to clean up 10s of GB of these extra directories, but their system is not so great at deletions (plagued with timeouts and their poor browser based client that sends chatty messages back and forth for every object).

        Conscious tech

        girishG 1 Reply Last reply
        0
        • robiR robi

          @girish when backups fail and retry, it seems to cause this as evidenced by multiple "directories" in the object store of the SAME date and different session strings.

          So I wouldn't know if this is an SDK problem or an implementation of the SDK.

          I am attempting to clean up 10s of GB of these extra directories, but their system is not so great at deletions (plagued with timeouts and their poor browser based client that sends chatty messages back and forth for every object).

          girishG Do not disturb
          girishG Do not disturb
          girish
          Staff
          wrote on last edited by
          #4

          @robi Oh, seeing session strings in the UI seems definitely like some backend issue. I am not sure what the client can do about those. Did their support say anything about this?

          robiR 1 Reply Last reply
          0
          • girishG girish

            @robi Oh, seeing session strings in the UI seems definitely like some backend issue. I am not sure what the client can do about those. Did their support say anything about this?

            robiR Offline
            robiR Offline
            robi
            wrote on last edited by
            #5

            @girish no, they're pretty useless. Their web UI S3 console is such crap it can't handle the chatty API requests and keeps timing out. Also I may be wrong that multiple directories are because of failures and restarts. It just looks like multiple changed apps per day get a new dir.

            So I am attempting other workarounds. Like creating a new bucket and just nuking the old one.

            rsync isn't great for object store backups as it makes a ton of small files.
            tgz isn't great as it's a lot of repeated information.

            We need something hybrid that is the best of both.
            Something like backing up to a local Minio much more quickly then doing an object to object store transfer offsite, which is much more efficient. This may also offer an opportunity to dedupe and further optimize.

            Conscious tech

            1 Reply Last reply
            0
            Reply
            • Reply as topic
            Log in to reply
            • Oldest to Newest
            • Newest to Oldest
            • Most Votes


            • Login

            • Don't have an account? Register

            • Login or register to search.
            • First post
              Last post
            0
            • Categories
            • Recent
            • Tags
            • Popular
            • Bookmarks
            • Search