Solved Restoring file timestamps from backup / migration

[robi]

This is a fairly serious issue, since file content is time stamped and sorted at the filesystem level.

Restoring a backup doesn't seem to restore the original timestamps from the time of backup. (tgz)

Performing a storage migration also does not preserve the timestamps for older content. (cp)

What parameters are being used for cp migration & tgz restore from backup ?

How do we make this work?

[girish]

Generally, web apps don't care about filesystem timestamps. Is this because of surfer showing the filesystem timestamps?

[robi]

@girish Yes, it's also a file server for us.

[girish]

@robi For tgz restore, timestamps are restored for me. Can you check if the timestamps are stored correctly in the backup using tar -ztvf app_xx.tar.gz ? For example, in my case, I see:

-rw-r--r-- 0/0               0 2009-11-17 15:33 ./data/public/try

Then, after restore, I can confirm that the timestamp is the same. Same for the data migration, after the migrate, the timestamp is preserved:

root@vultr:/mnt/surferdata/public# ls -lh
total 4.0K
-rw-r--r-- 1 yellowtent yellowtent    0 Nov 17  2009 try

[girish]

It is possible you are using rsync and not tgz? If so, yes, timestamps are not restored for rsync case. I guess we need a mechanism to save all the timestamps in some meta file and restore them back.

[robi]

It appears the backup was changed to rsync, and the FTP client config wasn't preserving timestamps when missing content was reuploaded.

We're ok for now, in this use case.

rsync could be 'fixed' by using .tar before the rsync.
Could be made more efficient too if a fixed chunk size or stream was used, so all the small file operations don't need to be done.

[girish]

@robi there is a already bunch of "workarounds" for rsync. Empty directories, executable bit of files cannot be stored in most object storage. So, there is fsmetadata.json file that stores this information outside of the files. When restoring, we use that file to restore back the state. I guess we can extend that file to also save and restore timestamps.

If anyone wants this leave a note and I can look into it in the future.