Run s3_media_upload script
-
@girish Thank you for responding!
Yes, this thread is about the script that you have linked (https://github.com/matrix-org/synapse-s3-storage-provider#regular-cleanup-job). It is part of Cloudron's Synapse installation and can be found in /app/code/env/bin.
I had already managed to make the database config as you have mentioned in your post.
The problem is: The script uses "Boto3" (AWS SDK for Python) which expects the S3 credentials either to be saved in the config file ~/.aws/credentials or as environment variables, see
https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.htmlPlease correct me if I'm wrong, but Cloudron doesn't grant me access to either of these. That's why I mentioned you in this thread. I think you'd have to enable one of these options to make the script work.
-
@nichu42 From the link you posted, there is a bunch of environment variables you can set - both for the credentials itself and also for the config file. Have you tried those? Or is the question about how to use those env variables?
-
@nichu42 You are running this on a Web Terminal right ? You can just
export foo=barlike in a normal terminal and then run the s3_media_upload script ?@girish Yay! Thank you.
I am all new to this Linux game so I wasn't aware I could just set the environment variables like that.For everyone else, this is what you need to do:
──────────────────────────────
-
Set up S3 with Synapse. See my post here: https://forum.cloudron.io/post/60415
-
Create a database.yaml file in /app/data/configs that contains the postgres database credentials.
You can find those in the existing homeserver.yaml file.
user: xxx password: xxx database: xxx host: postgresql- Create a script (e.g., s3cleanup.sh) with the following contents:
#!/bin/bash cd /app/data/configs export AWS_ACCESS_KEY_ID=[your S3 compatible access key] export AWS_SECRET_ACCESS_KEY=[your s3 compatible secret access key] /app/code/env/bin/s3_media_upload update /app/data/data/media_store 1m /app/code/env/bin/s3_media_upload upload --delete --endpoint-url https://yours3storageendpoint.com /app/data/data/media_store [your s3_bucket_name]- Run the s3cleanup.sh script.
It will look up media that hasn't been touched for 1m (= 1 month) or whatever you set above. It needs to be an integeger value, followed by either m = month(s), d = day(s) or y = year(s).
It will create a cache.db file that refers to the media that matches your criteria.
In the second step, it will upload all files from the cache.db to your s3 storage and delete the local copies.
The output looks like this:
Syncing files that haven't been accessed since: 2022-12-25 14:59:14.674154 Synced 603 new rows 100%|████████████████████████████████████| 603/603 [00:00<00:00, 16121.24files/s] Updated 0 as deleted 100%|████████████████████████████████████| 603/603 [03:25<00:00, 2.93files/s] Uploaded 603 media out of 603 Uploaded 3203 files Uploaded 263.6M Deleted 603 media Deleted 3203 files Deleted 263.6MEdit: Added path /app/data/configs to script to make it work as cron job.
Edit2: Added more choices for duration suffixes in 's3_media_upload update' job.Disclaimer: This is to the best of my knowledge and understanding. It worked for me, but I accept no liability for loss of data on your server caused by my incompetence.
Matrix: @nichu42:blueplanet.social
-
-
@girish Yay! Thank you.
I am all new to this Linux game so I wasn't aware I could just set the environment variables like that.For everyone else, this is what you need to do:
──────────────────────────────
-
Set up S3 with Synapse. See my post here: https://forum.cloudron.io/post/60415
-
Create a database.yaml file in /app/data/configs that contains the postgres database credentials.
You can find those in the existing homeserver.yaml file.
user: xxx password: xxx database: xxx host: postgresql- Create a script (e.g., s3cleanup.sh) with the following contents:
#!/bin/bash cd /app/data/configs export AWS_ACCESS_KEY_ID=[your S3 compatible access key] export AWS_SECRET_ACCESS_KEY=[your s3 compatible secret access key] /app/code/env/bin/s3_media_upload update /app/data/data/media_store 1m /app/code/env/bin/s3_media_upload upload --delete --endpoint-url https://yours3storageendpoint.com /app/data/data/media_store [your s3_bucket_name]- Run the s3cleanup.sh script.
It will look up media that hasn't been touched for 1m (= 1 month) or whatever you set above. It needs to be an integeger value, followed by either m = month(s), d = day(s) or y = year(s).
It will create a cache.db file that refers to the media that matches your criteria.
In the second step, it will upload all files from the cache.db to your s3 storage and delete the local copies.
The output looks like this:
Syncing files that haven't been accessed since: 2022-12-25 14:59:14.674154 Synced 603 new rows 100%|████████████████████████████████████| 603/603 [00:00<00:00, 16121.24files/s] Updated 0 as deleted 100%|████████████████████████████████████| 603/603 [03:25<00:00, 2.93files/s] Uploaded 603 media out of 603 Uploaded 3203 files Uploaded 263.6M Deleted 603 media Deleted 3203 files Deleted 263.6MEdit: Added path /app/data/configs to script to make it work as cron job.
Edit2: Added more choices for duration suffixes in 's3_media_upload update' job.Disclaimer: This is to the best of my knowledge and understanding. It worked for me, but I accept no liability for loss of data on your server caused by my incompetence.
@nichu42 said in Run s3_media_upload script:
Run the s3cleanup.sh script
How do I run this script?
When I callbash /app/data/s3cleanup.sh, I get the following output:/app/data/s3cleanup.sh: line 2: cd: $'/app/data/configs\r': No such file or directory usage: s3_media_upload update [-h] base_path duration s3_media_upload update: error: argument duration: duration must be an integer followed by a 'd', 'm' or 'y' suffix usage: s3_media_upload [-h] [--no-progress] {update-db,check-deleted,update,write,upload} ... s3_media_upload: error: Could not open 'cache.db' as sqlite DB: unable to open database file -
-
@nichu42 said in Run s3_media_upload script:
Run the s3cleanup.sh script
How do I run this script?
When I callbash /app/data/s3cleanup.sh, I get the following output:/app/data/s3cleanup.sh: line 2: cd: $'/app/data/configs\r': No such file or directory usage: s3_media_upload update [-h] base_path duration s3_media_upload update: error: argument duration: duration must be an integer followed by a 'd', 'm' or 'y' suffix usage: s3_media_upload [-h] [--no-progress] {update-db,check-deleted,update,write,upload} ... s3_media_upload: error: Could not open 'cache.db' as sqlite DB: unable to open database file@WiseMetalhead said in Run s3_media_upload script:
How do I run this script?
When I callbash /app/data/s3cleanup.sh, I get the following output:/app/data/s3cleanup.sh: line 2: cd: $'/app/data/configs\r': No such file or directoryIt seems your script uses CR+LF where it should only have the LF end of line character (resulting in the tailing \r at the end of the path, which makes it invalid).
Are you using Windows? If so, I recommend using Notepad++ (https://notepad-plus-plus.org/).
Open your script, then select Edit > EOL Conversion > Unix (LF). Save, upload and try again.Matrix: @nichu42:blueplanet.social
-
@WiseMetalhead said in Run s3_media_upload script:
How do I run this script?
When I callbash /app/data/s3cleanup.sh, I get the following output:/app/data/s3cleanup.sh: line 2: cd: $'/app/data/configs\r': No such file or directoryIt seems your script uses CR+LF where it should only have the LF end of line character (resulting in the tailing \r at the end of the path, which makes it invalid).
Are you using Windows? If so, I recommend using Notepad++ (https://notepad-plus-plus.org/).
Open your script, then select Edit > EOL Conversion > Unix (LF). Save, upload and try again.@nichu42 said in Run s3_media_upload script:
Open your script, then select Edit > EOL Conversion > Unix (LF). Save, upload and try again.
It actually works. Thank you!
Strangely enough, the script was originally created using the Cloudron file manager.
-
@WiseMetalhead @nichu42 Sorry to open this up again but I'm interested in setting synapse up with s3 and am curious as to how your experience has been so far
-
@WiseMetalhead @nichu42 Sorry to open this up again but I'm interested in setting synapse up with s3 and am curious as to how your experience has been so far
-
@andreasdueren Still running it as described above Everything is fine.
@nichu42 OK thanks I have to read through the whole documentation then to try this out
