SOLVED Cannot migrate app

[robi]

After some Surfer escapades, the original App and it's install location was deleted and a new one was cloned to test the fixes.

Now the new one is not where it should be and needs migrating back to the original mount point location.

The issue is that the process crashes at around the 20GB mark (it's 10x that) and the resume task doesn't actually complete anything.

I'll email the log in, but I haven't found anything useful in it ;-/

If you're reading this and can validate, please test migrating a ~50GB app if you can.

[robi]

Attempting again with increased memory limits..

Turns out one cannot set the memory limit to the max of the allowed bar. It errors. It only accepted at 1G less.

Perhaps a good feature request here is to check on certain operations and the likely memory requirements before starting the task.

This issue came up before where the copy/migration/backup process is so memory leaky or a hog that won't release previously copied data until all of it is complete.
This makes migration of large data stores impossible.

I gave my Surfer app 31GB or RAM as the highest limit and it still failed.

Help!

[girish]

@robi Sent you a mail from support, it seems the app is running now. Did you figure out what the issue was?

[robi]

@girish No, still needs attention, see email reply.

[girish]

Thanks, the task was running out of memory because internally memory limit for data migration is set to 400MB. It seems cp takes lot more memory than I thought when copying large files. I bumped it up to 1024M and it works. I have fixed this in next release.

[robi]

@girish This reminds me of the old DOS days where copy vs xcopy was a big deal.

Is there an equivalent to xcopy in Linux?

[atridad]

@girish Perhaps a slider like in the services section with the default at 1gb?

[robi]

That also means that the Task resume for a failure like this is broken.

One could copy the rest of the files if the resume skipped existing files.

Ideally the copy would be more memory efficient and release memory after a threshold.

Otherwise one could set up a local socket and slurp it over. This would have the additional benefit of working across systems in Multi-Cloudron.

[girish]

@atridad yeah, I just bumped the internal default for data directory migration to be 1GB . I tested with ~500GB now and it's all good.