Solved How to rsync files into surfer

[jdaviescoates]

My friends who've been hosting library.uniteddiversity.coop for years want to downscale their server and so I need to find it a new home... My plan is to just use Surfer, but I'm struggling to work out how to rsync my files?

I've been trying locally mounting Surfer using Davfs to /home/josef/Library2/ and then doing this:

rsync -avrh josef@library.uniteddiversity.coop:/data/josef/library.uniteddiversity.coop/ /home/josef/Library2/

But it doesn't have the permissions to mkdir.

I can SSH into library.uniteddiversity.coop

I'm guess there must be a simple way to do this but I'm not sure what it is?!? Help!

Thanks,

Josef.

[nebulon]

I think there are two things here. Rsync as such would only work via the SSH connection into the host system. For that you need to find the surfer instance data folder. Unless changed, this is at /home/yellowtent/appsdata/<appid>/data/public

For the webdav mount, creating directories should work fine, I use a webdav mount (although via gnome filemanager) all the time. Do you have any logs or how did you setup the mountpoint?

[jdaviescoates]

@nebulon said in How to rsync files into surfer:

I think there are two things here. Rsync as such would only work via the SSH connection into the host system. For that you need to find the surfer instance data folder. Unless changed, this is at /home/yellowtent/appsdata/<appid>/data/public

Hmz, OK, so I've found the appid

But now I'm unsure of what command I should try.

I just tried:

rsync -avrh josef@library.uniteddiversity.coop:/data/josef/library.uniteddiversity.coop/ jdaviescoates@library2.uniteddiversity.com:/home/yellowtent/appsdata/e815de83-e018-4507-be2d-e3d77d818aae/data/public

But got:

The source and destination cannot both be remote.

So I guess I need to SSH into library.uniteddiversity.coop and then run the command there...

So I tried:

josef@library:~$ rsync -avrhP /data/josef/library.uniteddiversity.coop/ jdaviescoates@library2.uniteddiversity.coop:/home/yellowtent/appsdata/e815de83-e018-4507-be2d-e3d77d818aae/data/public
The authenticity of host 'library2.uniteddiversity.coop (116.203.252.254)' can't be established.
ECDSA key fingerprint is f4:12:64:cd:e0:bd:2e:e3:39:24:ad:17:98:13:c2:a2.
Are you sure you want to continue connecting (yes/no)? yes 
Warning: Permanently added 'library2.uniteddiversity.coop,116.203.252.254' (ECDSA) to the list of known hosts.
jdaviescoates@library2.uniteddiversity.coop's password: 
Permission denied, please try again.

Seems like I'm getting there, but for some reason it's not accepting my pw (I'm copy/ pasting of bitwarden, but maybe the pasting isn't working or something?)

@nebulon said in How to rsync files into surfer:

For the webdav mount, creating directories should work fine, I use a webdav mount (although via gnome filemanager) all the time. Do you have any logs or how did you setup the mountpoint?

At first I just went to + Other Location and used the Connect to Server field and the bottom of the screen.

Then I just dragged of the files from one gnome filemanager tab to the other. This seemed to mostly work fine aside from the fact I kept getting those symlink errors, and then some other errors too, like e.g.

It was because I kept getting various errors (which stopped the copying process until I chose Skip) that I decided to try the rsync route.

To try that I decided to try mounting the Surfer app using:

sudo mount -t davfs https://library2.uniteddiversity.coop/_webdav/ /home/josef/Library2/

I think I tried doing it without sudo first but it wouldn't let me.

But I've realised that is why I can't mkdir, because /home/josef/Library2/ is owned by root.

So I just ran:

sudo chown -R josef /home/josef/Library2/

Which seemed to fix that issue, although I did also get this:

chown: changing ownership of '/home/josef/Library2/lost+found': Invalid argument

Anyways, I ignored that for now and tried my previous rsync command:

rsync -avrh josef@library.uniteddiversity.coop:/data/josef/library.uniteddiversity.coop/ /home/josef/Library2/

And this now seems to be working!

Although it is still having issues with the symlinks:

rsync: symlink "/home/josef/Library2/Community_Currencies" -> "/data/josef/library.uniteddiversity.coop/Money_and_Economics/Community_Currencies" failed: Function not implemented (38)
rsync: symlink "/home/josef/Library2/Cooperatives" -> "/data/josef/library.uniteddiversity.coop/Money_and_Economics/Cooperatives" failed: Function not implemented (38)

(and I forgot to add P to the -avrh to show me the progress and help restart partial it something happens mid-way)...

So I seem to managed to do this now - yay!

But there is still the symlink issue.

[jeau]

On the other hand, is ther a way to run a cron job inside a Surfer app?

[nebulon]

@jeau there is no way to do so currently. What would be the use-case for that?

[jeau]

@nebulon I have an ongoing project with two requests from the team:

• automatically generate a catalog from remote files,
• get an updated static site on github

I can easily make a script to do this inside the Surfer app if this script can be launched periodically

[thetomester13]

@jeau I think a way to do this would be to have a cron running on a different system (even on a LAMP app maybe) and use the surfer CLI to push files.

[nebulon]

@thetomester13 right I also think this does not directly belong to the surfer app. It could maybe also more easily achieved through some git hooks or ci/cd pipelines.

[jeau]

@nebulon @thetomester13 I think you're right, there are several ways to do it