Latest attempt - why oh why does this just not work????

[scooke]

Hi everyone,
I thought I'd take another stab at getting my Peertube to use Object Storage, using a Cloudron-installed Minio instance (on a different server). Here is what I have in my production.yaml:

object_storage:
  enabled: true
  driver: 'minio'
  pathstyle: 'true'
  endpoint: 'https://minioapi.server.com' (it used to be minio-api... but I removed the - like another poster did, and then at least the upload part would start. Before, not always, as I messed with different settings.)
  region: 'ca-central-1'
  upload_acl:
    # Set this ACL on each uploaded object of public/unlisted videos
    public: 'public'
    # Set this ACL on each uploaded object of private/internal videos
    # PeerTube can proxify requests to private objects so your users can access them
    private: 'private'
  protocol: 'HTTPS'
  max_upload_part: '2GB'
  web_videos:
    bucketName: 'vids'
    prefix: 'web/'
  # Use the same bucket as for web videos but with a different prefix
  streaming_playlists:
    bucketName: 'vids'
    prefix: 'streaming/'
credentials:
  access_key_id: 'longkey'
  secret_access_key: 'longerkey'

ar 31 16:31:09 [example.com:443] 2024-03-31 14:31:09.337 info: Creating external storage move job for video 1b4a83e8-c474-4b28-bcf2-4ccacadc15e1.
Mar 31 16:31:09 [example.com:443] 2024-03-31 14:31:09.352 info: Moving video 1b4a83e8-c474-4b28-bcf2-4ccacadc15e1 to object storage in job 13.
Mar 31 16:31:09 [example.com:443] 2024-03-31 14:31:09.957 info: Initialized S3 client https://minioapi.server.com with region ca-central-1.
Mar 31 16:31:09 [example.com:443] 2024-03-31 14:31:09.968 info: Processing transcoding job builder in job af6ed39e-1e1f-40e4-af70-20b84f2c2a1d.
Mar 31 16:31:10 172.18.0.1 - - [31/Mar/2024:14:31:10 +0000] "GET / HTTP/1.1" 200 6492 "-" "Mozilla (CloudronHealth)"
Mar 31 16:31:10 [example.com:443] 2024-03-31 14:31:10.282 info: 172.18.0.1 - - [31/Mar/2024:14:31:10 +0000] "GET / HTTP/1.0" 200 18062 "-" "Mozilla (CloudronHealth)"
Mar 31 16:31:12 [example.com:443] 2024-03-31 14:31:12.270 error: Cannot move video https://example.com/videos/watch/1b4a83e8-c474-4b28-bcf2-4ccacadc15e1 storage. {
Mar 31 16:31:12 "err": {
Mar 31 16:31:12 "stack": "CredentialsProviderError: Could not load credentials from any providers\n at /app/code/server/node_modules/@aws-sdk/credential-provider-node/dist-cjs/defaultProvider.js:13:11\n at /app/code/server/node_modules/@smithy/property-provider/dist-cjs/chain.js:12:39\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async coalesceProvider (/app/code/server/node_modules/@smithy/property-provider/dist-cjs/memoize.js:14:24)\n at async SignatureV4.credentialProvider (/app/code/server/node_modules/@smithy/property-provider/dist-cjs/memoize.js:33:24)\n at async SignatureV4.signRequest (/app/code/server/node_modules/@smithy/signature-v4/dist-cjs/SignatureV4.js:106:29)\n at async /app/code/server/node_modules/@aws-sdk/middleware-signing/dist-cjs/awsAuthMiddleware.js:16:18\n at async /app/code/server/node_modules/@smithy/middleware-retry/dist-cjs/retryMiddleware.js:27:46\n at async /app/code/server/node_modules/@aws-sdk/middleware-flexible-checksums/dist-cjs/flexibleChecksumsMiddleware.js:57:20\n at async /app/code/server/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/region-redirect-endpoint-middleware.js:14:24\n at async /app/code/server/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/region-redirect-middleware.js:9:20\n at async /app/code/server/node_modules/@aws-sdk/middleware-logger/dist-cjs/loggerMiddleware.js:7:26\n at async Promise.all (index 0)\n at async Upload.__uploadUsingPut (/app/code/server/node_modules/@aws-sdk/lib-storage/dist-cjs/Upload.js:66:26)\n at async Upload.__doConcurrentUpload (/app/code/server/node_modules/@aws-sdk/lib-storage/dist-cjs/Upload.js:128:28)\n at async Promise.all (index 0)\n at async Upload.__doMultipartUpload (/app/code/server/node_modules/@aws-sdk/lib-storage/dist-cjs/Upload.js:211:9)\n at async Upload.done (/app/code/server/node_modules/@aws-sdk/lib-storage/dist-cjs/Upload.js:39:16)\n at async uploadToStorage (file:///app/code/server/dist/core/lib/object-storage/shared/object-storage-helpers.js:152:23)\n at async moveWebVideoFiles (file:///app/code/server/dist/core/lib/job-queue/handlers/move-to-object-storage.js:39:25)\n at async moveToJob (file:///app/code/server/dist/core/lib/job-queue/handlers/shared/move-video.js:19:13)\n at async Object.processMoveToObjectStorage [as move-to-object-storage] (file:///app/code/server/dist/core/lib/job-queue/handlers/move-to-object-storage.js:16:5)\n at async Object.wrapPromiseFun (file:///app/code/server/dist/core/lib/plugins/hooks.js:8:24)\n at async Worker.processJob (/app/code/server/node_modules/bullmq/dist/cjs/classes/worker.js:350:28)\n at async Worker.retryIfFailed (/app/code/server/node_modules/bullmq/dist/cjs/classes/worker.js:535:24)",
Mar 31 16:31:12 "message": "Could not load credentials from any providers",
Mar 31 16:31:12 "tryNextLink": false,
Mar 31 16:31:12 "name": "CredentialsProviderError",
Mar 31 16:31:12 "$metadata": {
Mar 31 16:31:12 "attempts": 1,
Mar 31 16:31:12 "totalRetryDelay": 0
Mar 31 16:31:12 }
Mar 31 16:31:12 }
Mar 31 16:31:12 }
Mar 31 16:31:12 [example.com:443] 2024-03-31 14:31:12.287 error: Cannot execute job 13 in queue move-to-object-storage. {
Mar 31 16:31:12 "payload": {
Mar 31 16:31:12 "videoUUID": "1b4a83e8-c474-4b28-bcf2-4ccacadc15e1",
Mar 31 16:31:12 "isNewVideo": true,
Mar 31 16:31:12 "previousVideoState": 2
Mar 31 16:31:12 },
Mar 31 16:31:12 "err": {
Mar 31 16:31:12 "stack": "CredentialsProviderError: Could not load credentials from any providers\n at /app/code/server/node_modules/@aws-sdk/credential-provider-node/dist-cjs/defaultProvider.js:13:11\n at /app/code/server/node_modules/@smithy/property-provider/dist-cjs/chain.js:12:39\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async coalesceProvider (/app/code/server/node_modules/@smithy/property-provider/dist-cjs/memoize.js:14:24)\n at async SignatureV4.credentialProvider (/app/code/server/node_modules/@smithy/property-provider/dist-cjs/memoize.js:33:24)\n at async SignatureV4.signRequest (/app/code/server/node_modules/@smithy/signature-v4/dist-cjs/SignatureV4.js:106:29)\n at async /app/code/server/node_modules/@aws-sdk/middleware-signing/dist-cjs/awsAuthMiddleware.js:16:18\n at async /app/code/server/node_modules/@smithy/middleware-retry/dist-cjs/retryMiddleware.js:27:46\n at async /app/code/server/node_modules/@aws-sdk/middleware-flexible-checksums/dist-cjs/flexibleChecksumsMiddleware.js:57:20\n at async /app/code/server/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/region-redirect-endpoint-middleware.js:14:24\n at async /app/code/server/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/region-redirect-middleware.js:9:20\n at async /app/code/server/node_modules/@aws-sdk/middleware-logger/dist-cjs/loggerMiddleware.js:7:26\n at async Promise.all (index 0)\n at async Upload.__uploadUsingPut (/app/code/server/node_modules/@aws-sdk/lib-storage/dist-cjs/Upload.js:66:26)\n at async Upload.__doConcurrentUpload (/app/code/server/node_modules/@aws-sdk/lib-storage/dist-cjs/Upload.js:128:28)\n at async Promise.all (index 0)\n at async Upload.__doMultipartUpload (/app/code/server/node_modules/@aws-sdk/lib-storage/dist-cjs/Upload.js:211:9)\n at async Upload.done (/app/code/server/node_modules/@aws-sdk/lib-storage/dist-cjs/Upload.js:39:16)\n at async uploadToStorage (file:///app/code/server/dist/core/lib/object-storage/shared/object-storage-helpers.js:152:23)\n at async moveWebVideoFiles (file:///app/code/server/dist/core/lib/job-queue/handlers/move-to-object-storage.js:39:25)\n at async moveToJob (file:///app/code/server/dist/core/lib/job-queue/handlers/shared/move-video.js:19:13)\n at async Object.processMoveToObjectStorage [as move-to-object-storage] (file:///app/code/server/dist/core/lib/job-queue/handlers/move-to-object-storage.js:16:5)\n at async Object.wrapPromiseFun (file:///app/code/server/dist/core/lib/plugins/hooks.js:8:24)\n at async Worker.processJob (/app/code/server/node_modules/bullmq/dist/cjs/classes/worker.js:350:28)\n at async Worker.retryIfFailed (/app/code/server/node_modules/bullmq/dist/cjs/classes/worker.js:535:24)",
Mar 31 16:31:12 "message": "Could not load credentials from any providers",
Mar 31 16:31:12 "tryNextLink": false,
Mar 31 16:31:12 "name": "CredentialsProviderError",
Mar 31 16:31:12 "$metadata": {
Mar 31 16:31:12 "attempts": 1,
Mar 31 16:31:12 "totalRetryDelay": 0
Mar 31 16:31:12 }

The main problem seems to be CredentialsProviderError: Could not load credentials from any providers\n

Now, is this just because I don't have the exactly correct syntax? What would that be? What gets capitalized, what gets '', what gets indented and how many spaces? Or have I missed some settings on the bucket? It's public with read-write settings, and at least one user who is listed as having access. Or is it CORS??? I thought for sure this would work with a Cloudron-installed Minio.

If you DO have a working Cloudron-based Peertube successfully linked to a Minio instance, perhaps you could share your .env?

[jdaviescoates]

@scooke said in Latest attempt - why oh why does this just not work????:

If you DO have a working Cloudron-based Peertube successfully linked to a Minio instance, perhaps you could share your .env?

I've not tried it with a Cloudron Minio yet, but I do have it working with Scaleway Object Storage.

This is what I've got (which I now see appears to have auto-changed - presumably during some update - since I actually did it):

object_storage:
  enabled: true
  # Scaleway endpoint
  endpoint: 's3.fr-par.scw.cloud'
  region: 'fr-par'
  videos:
    bucket_name: 'bucket-name'
    prefix: 'videos/'
  # Use the same bucket as for webtorrent videos but with a different prefix
  streaming_playlists:
    bucket_name: 'bucket-name'
    prefix: 'streaming-playlists/'
  credentials:
    access_key_id: 'XXX'
    secret_access_key: 'XXX'

[scooke]

@jdaviescoates I might just move on over to Scaleway too then. Thanks.

[jdaviescoates]

@scooke you could also experiment with taking off the https:// from your endpoint as I don't think that's supposed to be there