How to run these Peertube CLI commands to move videos to object storage on Cloudron?

[jdaviescoates]

According to the Peertube docs at https://docs.joinpeertube.org/maintain/tools#move-video-files-to-object-storage I should be able to use the Peertube CLI to move my videos to object storage (I just imported a whole YouTube channel before realising I hadn't set object storage up )

But I can't work out how to get it to work via the Cloudron web terminal.

Can @staff help? Thanks in advance

[girish]

@jdaviescoates in the past https://docs.cloudron.io/apps/peertube/#cli used to work. Does that not work?

[jdaviescoates]

@girish said in How to run these Peertube CLI commands to move videos to object storage on Cloudron?:

@jdaviescoates in the past https://docs.cloudron.io/apps/peertube/#cli used to work. Does that not work?

I'd guess those specific commands (and others too, most likely) work, but I've tried loads of versions of the lines of code shown in those Peertube docs but nothing works.

I mostly just get command not found type errors.

[girish]

@jdaviescoates Can you try with gosu cloudron:cloudron npm run create-move-video-storage-job <rest-of-args> ?

[jdaviescoates]

@girish said in How to run these Peertube CLI commands to move videos to object storage on Cloudron?:

@jdaviescoates Can you try with gosu cloudron:cloudron npm run create-move-video-storage-job <rest-of-args> ?

This seemed to do the trick!

gosu cloudron:cloudron npm run create-move-video-storage-job -- --to-object-storage --all-videos

Thank you!

Now just have to wait to see if it actually works and the videos remain playable...

But from what I can tell so far it appears to be working (in that the S3 bucket that was previously empty is now beginning to fill up with stuff! )

[jdaviescoates]

@jdaviescoates said in How to run these Peertube CLI commands to move videos to object storage on Cloudron?:

Now just have to wait to see if it actually works and the videos remain playable...

But from what I can tell so far it appears to be working (in that the S3 bucket that was previously empty is now beginning to fill up with stuff! )

Nope, now none of the videos will play

Seeing errors like this:

Oct 06 16:54:11[uniteddiversity.tv:443] 2023-10-06 15:54:11.837 error: Client log: HLS.js error: networkError - fatal: true - manifestLoadError {
Oct 06 16:54:11"username": "uniteddiversity",
Oct 06 16:54:11"userAgent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/117.0",
Oct 06 16:54:11"url": "https://uniteddiversity.tv/w/ag8ap5Akunqki5jgtvdymb"
Oct 06 16:54:11}

[jdaviescoates]

this has evidently got something to do with CORS Rule not working, but I'm at a total loss as to why, as I've done the exact same thing that made it all work on bridport.tv but it just won't work on uniteddiversity.tv

They literally have the exact same CORS rules, but one works and one doesn't

josef@josef-ThinkPad-P50:~$ aws s3api get-bucket-cors --bucket bridport-tv
{
    "CORSRules": [
        {
            "AllowedHeaders": [
                "*"
            ],
            "AllowedMethods": [
                "GET",
                "HEAD",
                "POST",
                "PUT",
                "DELETE"
            ],
            "AllowedOrigins": [
                "https://bridport.tv",
                "https://uniteddiversity.tv"
            ],
            "ExposeHeaders": [
                "Etag"
            ],
            "MaxAgeSeconds": 3000
        }
    ]
}
josef@josef-ThinkPad-P50:~$ aws s3api get-bucket-cors --bucket uniteddiversity-tv
{
    "CORSRules": [
        {
            "AllowedHeaders": [
                "*"
            ],
            "AllowedMethods": [
                "GET",
                "HEAD",
                "POST",
                "PUT",
                "DELETE"
            ],
            "AllowedOrigins": [
                "https://bridport.tv",
                "https://uniteddiversity.tv"
            ],
            "ExposeHeaders": [
                "Etag"
            ],
            "MaxAgeSeconds": 3000
        }
    ]
}
josef@josef-ThinkPad-P50:~$ curl -X OPTIONS -H 'Origin: https://bridport.tv' https://bridport-tv.s3.fr-par.scw.cloud/index.html -H "Access-Control-Request-Method: GET"
josef@josef-ThinkPad-P50:~$ curl -X OPTIONS -H 'Origin: https://uniteddiversity.tv' https://uniteddiversity-tv.s3.fr-par.scw.cloud/index.html -H "Access-Control-Request-Method: GET"

CORSForbiddenCORSResponse: This CORS request is not allowed. This is usually because the evalution of Origin, request method / Access-Control-Request-Method or Access-Control-Request-Headers are not whitelisted by the resource's CORS spec.txbfe095d4026b4523bd3c9-00652044f0BUCKETGET
josef@josef-ThinkPad-P50:~$

/me bangs my head against the wall

[girish]

@jdaviescoates This seems like maybe the bucket configuration is not getting applied on scaleway, no?

I can atleast confirm the CORS configuration looks exact same .

[jdaviescoates]

Thankfully I eventually managed to resolve this!

After a week of not really getting anywhere with Scaleway support they eventually told me to try adding --debug to my commands! And just doing that gave me the clue I needed to sort it out!

I spotted this:

2023-10-14 22:50:58,821 - MainThread - botocore.regions - DEBUG - Endpoint provider result: https://bridport-tv.s3.fr-par.scw.cloud/uniteddiversity-tv

And that endpoint was wrong.

What was happening was that the [profile two] stuff I was trying to do as per https://www.scaleway.com/en/docs/storage/object/api-cli/object-storage-aws-cli/ didn't seem to be working and it was instead trying to use the [default] profile, resulting in the wrong endpoint.

I just edited the [default] to have the correct details for the uniteddiversity-tv bucket and then it worked! Not sure why I didn't think to trying doing that before!

[chymian]

@girish said in How to run these Peertube CLI commands to move videos to object storage on Cloudron?:

@jdaviescoates Can you try with gosu cloudron:cloudron npm run create-move-video-storage-job <rest-of-args> ?

that cmd is not working on my installation.

root@my /home/yellowtent/appsdata/03a27d2a-5afc-4bed-9913-2e7ad6fbd978/data
$ gosu yellowtent:yellowtent npm run create-move-video-storage-job -- --to-object-storage --all-videos
npm ERR! code ENOENT
npm ERR! syscall open
npm ERR! path /home/yellowtent/appsdata/03a27d2a-5afc-4bed-9913-2e7ad6fbd978/data/package.json
npm ERR! errno -2
npm ERR! enoent ENOENT: no such file or directory, open '/home/yellowtent/appsdata/npm_***/data/package.json'
npm ERR! enoent This is related to npm not being able to find a file.
npm ERR! enoent 

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/yellowtent/.npm/_logs/2023-10-23T09_12_24_838Z-debug-0.log
root@my /home/yellowtent/appsdata/03a27d2a-5afc-4bed-9913-2e7ad6fbd978/data
$ ls
local-production.json  production2.yaml  production.yaml  production.yaml.org  storage

from the log file:

…
25 verbose logfile logs-max:10 dir:/home/yellowtent/.npm/_logs/2023-10-23T09_09_04_886Z-
26 verbose logfile /home/yellowtent/.npm/_logs/2023-10-23T09_09_04_886Z-debug-0.log
27 timing npm:load:logFile Completed in 8ms
28 timing npm:load:timers Completed in 0ms
29 timing npm:load:configScope Completed in 0ms
30 timing npm:load Completed in 44ms
31 silly logfile done cleaning log files
32 timing command:run Completed in 18ms
33 verbose stack Error: ENOENT: no such file or directory, open '/home/yellowtent/appsdata/npm_***/data/package.json'
34 verbose cwd /home/yellowtent/appsdata/03a27d2a-5afc-4bed-9913-2e7ad6fbd978/data
35 verbose Linux 5.15.0-83-generic
36 verbose node v18.16.0
37 verbose npm  v9.5.1
38 error code ENOENT
39 error syscall open
40 error path /home/yellowtent/appsdata/03a27d2a-5afc-4bed-9913-2e7ad6fbd978/data/package.json
41 error errno -2
42 error enoent ENOENT: no such file or directory, open '/home/yellowtent/appsdata/npm_***/data/package.json'
43 error enoent This is related to npm not being able to find a file.
43 error enoent
44 verbose exit -2
45 timing npm Completed in 92ms
46 verbose code -2

[jdaviescoates]

@chymian said in How to run these Peertube CLI commands to move videos to object storage on Cloudron?:

that cmd is not working on my installation.

That's because you're doing it in the wrong place

You have to use the web terminal of your Peertube app within Cloudron.

As a general rule on Cloudron, except in rare occasions you don't/ shouldn't really ever have to login to the server to do anything.

[chymian]

sure,

I was context switching between 2 tasks and missed it completly.

thank you for your timly response.

[LoudLemur]

Thanks for this thread.

When Peertube v6 is available, I would like to try something similar. Cloudron documentation is good, but I feel it could be considerably improved in places by including an animated gif of the desktop showing how it happens.

[robi]

@LoudLemur If you contribute one, it can be added.

[LoudLemur]

@robi said in How to run these Peertube CLI commands to move videos to object storage on Cloudron?:

@LoudLemur If you contribute one, it can be added.

I suppose I could make a video for it, but it would be more like a Laurel and Hardy "How not to..." than a "How to...", I think! Custard pies, planks of wood and banana skins everywhere!