Docker Error 500 - Unable to pull image on same instance

[timconsidine]

Hmmm
I am now getting this error also

App is being installed.
=> Queued .
=> Registering subdomains
=> Downloading image ........................................
App installation error: Installation failed: Unable to pull image docker.domain.uk/cloudron-ocular:v1. registry error: {"reason":"server error","statusCode":500,"json":null}

But I can see the image in my docker registry so docker build and docker push is working.
The docker registry is on the Cloudron instance, but this has never been a problem before.

Have ensured docker login and cloudron install etc work

Running Cloudron v8.2.3 on Ubuntu 22.04

App install logs show :

box:docker pullImage: will pull docker.domain.uk/cloudron-ocular:v1. auth: no

Is that auth: no significant ?

Final log entry :

box:apptask run: app error for state pending_install: BoxError: Unable to pull image docker.domain.uk/cloudron-ocular:v1. registry error: {"reason":"server error","statusCode":500,"json":null} at pullImage (/home/yellowtent/box/src/docker.js:150:50) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async /home/yellowtent/box/src/docker.js:195:122 at async promiseRetry (/home/yellowtent/box/src/promise-retry.js:17:20) at async Object.downloadImage (/home/yellowtent/box/src/docker.js:194:5) at async downloadImage (/home/yellowtent/box/src/apptask.js:243:5) at async install (/home/yellowtent/box/src/apptask.js:337:5) { reason: 'Docker Error', details: {} }

box:tasks setCompleted - 22828: {"result":null,"error":{"stack":"BoxError: Unable to pull image docker.domain.uk/cloudron-ocular:v1. registry error: {"reason":"server error","statusCode":500,"json":null}\n at pullImage (/home/yellowtent/box/src/docker.js:150:50)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async /home/yellowtent/box/src/docker.js:195:122\n at async promiseRetry (/home/yellowtent/box/src/promise-retry.js:17:20)\n at async Object.downloadImage (/home/yellowtent/box/src/docker.js:194:5)\n at async downloadImage (/home/yellowtent/box/src/apptask.js:243:5)\n at async install (/home/yellowtent/box/src/apptask.js:337:5)","name":"BoxError","reason":"Docker Error","details":{},"message":"Unable to pull image docker.domain.uk/cloudron-ocular:v1. registry error: {"reason":"server error","statusCode":500,"json":null}"}}

box:tasks update 22828: {"percent":100,"result":null,"error":{"stack":"BoxError: Unable to pull image docker.domain.uk/cloudron-ocular:v1. registry error: {"reason":"server error","statusCode":500,"json":null}\n at pullImage (/home/yellowtent/box/src/docker.js:150:50)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async /home/yellowtent/box/src/docker.js:195:122\n at async promiseRetry (/home/yellowtent/box/src/promise-retry.js:17:20)\n at async Object.downloadImage (/home/yellowtent/box/src/docker.js:194:5)\n at async downloadImage (/home/yellowtent/box/src/apptask.js:243:5)\n at async install (/home/yellowtent/box/src/apptask.js:337:5)","name":"BoxError","reason":"Docker Error","details":{},"message":"Unable to pull image docker.domain.uk/cloudron-ocular:v1. registry error: {"reason":"server error","statusCode":500,"json":null}"}}

[nebulon]

Given that you see 500 status errors from your registry, can you check the logs there? 500 if used correctly refer to remote server errors. Auth failure would be in the 400 range (403, 402, ..)

[timconsidine]

In the docker registry app, logs show endless lines like this, but I don’t see errors as such :

Jan 23 11:41:20 127.0.0.1 - - [23/Jan/2025:11:41:20 +0000] "GET /v2 HTTP/1.0" 301 39 "" "Mozilla (CloudronHealth)"

Jan 23 11:41:20 time="2025-01-23T11:41:20.113222934Z" level=info msg="response completed" go.version=go1.20.8 http.request.host="localhost:5000" http.request.id=216d4214-238a-409b-930b-0a1ba5bf2f02 http.request.method=GET http.request.remoteaddr="127.0.0.1:56446" http.request.uri="/v2" http.request.useragent="Mozilla (CloudronHealth)" http.response.contenttype="text/html; charset=utf-8" http.response.duration="258.188µs" http.response.status=301 http.response.written=39

Jan 23 11:41:30 ::1 - - [23/Jan/2025:11:41:30 +0000] "GET /v2 HTTP/1.0" 301 39 "" "Mozilla (CloudronHealth)"

Jan 23 11:41:30 time="2025-01-23T11:41:30.129205782Z" level=info msg="response completed" go.version=go1.20.8 http.request.host="localhost:5000" http.request.id=ea05e778-e39b-47cd-8e12-5bb881512b3b http.request.method=GET http.request.remoteaddr="[::1]:54150" http.request.uri="/v2" http.request.useragent="Mozilla (CloudronHealth)" http.response.contenttype="text/html; charset=utf-8" http.response.duration="82.643µs" http.response.status=301 http.response.written=39

Jan 23 11:41:30 172.18.0.1 - - [23/Jan/2025:11:41:30 +0000] "GET /v2 HTTP/1.1" 301 39 "-" "Mozilla (CloudronHealth)”

Restarted the docker registry app on Cloudron, but same behaviour.
Image is visible in the registry app UI.

On local device I can do docker pull docker.domain.uk/image:tag so I guess the docker registry is ok, UI and images in app visible. And of course the docker push command worked.

[girish]

@timconsidine said in Docker Error 500 - Unable to pull image on same instance:

Is that auth: no significant ?

if the registry requires authentication then yes, this appears significant. Can you check the settings in Private Registry? Are you also able to docker pull on the cloudron itself?

[timconsidine]

@girish yes I can docker push and docker pull on my desktop so auth here is not a problem.
Also I have logged out and logged back in.
Cloudron CLI is able to create the app and sub-domain, so my cloudron auth is ok.
And I can do cloudron uninstall —app blah

[girish]

@timconsidine the cloudron cli auth is different from the registry auth . I guess you had done docker login at some point in your desktop and this is why push/pull works.

The code is supposed to get the authentication info from the Settings -> Private Docker Registry in this function - https://git.cloudron.io/platform/box/-/blob/master/src/docker.js?ref_type=heads#L112 . Have you configured Private Docker Registry? If so, maybe you can put a couple of console.log in that function to see what is going wrong. Maybe something wrong in the parsing logic

[timconsidine]

Thanks @girish
In a shipping container in DE, will have to investigate once I get back
Not least because I don’t understand about settings but will dive in

[timconsidine]

Right, back now, and I’ve re-read things, but am still not understanding.
Maybe time to explain like I’m a 5 year old

Docker registry is installed in Cloudron instance from the App store.
No special or changed settings for the registry.
Nothing in the app UI anyway to do so, and I haven’t (consciously) changed /app/data/config.yml and certainly not recently.
Been using this registry for custom app builds and never had a problem, no need to configure it.

the cloudron cli auth is different from the registry auth

Yes. I was attempting (badly) to say everything works, well clearly not everything.
I can use cloudron CLI on desktop to install and uninstall
I can use docker CLI on desktop to login / logout of the cloudron-installed registry
I can open the registry app in browser, and can see that it has the custom app image (pushed from desktop).
But cloudron still reports 500, cannot find image

I have 2FA on my cloudron login - I can login to the registry app in a browser without re-entering it, but maybe the cloudron install doesn’t have access to an authenticated login. Is it expecting 2FA ?

authentication info from the Settings -> Private Docker Registry

This has been set, and has never been an issue before.
But has something changed to require 2FA, or a user with no 2FA, or a token not password ?

[girish]

@timconsidine your setup seems correct. The 2FA aspect is indeed interesting! Maybe you put password into Private Docker Registry before you enabled 2FA? After you enabled 2FA, just the password is not enough to authenticate. So, you have to generate an App Password from your profile and use that in the Private Docker Registry settings.

[timconsidine]

@girish thank you
Yes, it’s entirely possible that I added 2FA subsequently.
I will try with an app password.

[girish]

@timconsidine said in Docker Error 500 - Unable to pull image on same instance:

box:docker pullImage: will pull docker.domain.uk/cloudron-ocular:v1. auth: no

Is that auth: no significant ?

2FA note aside, this still seems wrong. It should say auth: yes. The docker.domain.uk (from docker.domain.uk/cloudron-ocular:v1) must match the server address you put in Private Docker Registry UI.

[timconsidine]

Existing container registry :

• created an app password
• updated settings for private registry to use that
• desktop terminal : docker logout private registry then docker login with app password : success
• desktop terminal : cloudron logout then cloudron login (with 2FA)
• desktop terminal : cloudron install —image etc : failed

Jan 29 16:11:53 box:tasks update 22945: {"message":"Registering location app.somedomain.uk"}
Jan 29 16:11:53 box:dns upsertDnsRecords: subdomain:app domain:somedomain.uk type:A values:[“46.4.98.999"]
Jan 29 16:11:53 box:dns/manual upsert: app for zone somedomain.uk of type A with values ["46.4.98.999"]
Jan 29 16:11:53 box:dns upsertDnsRecords: subdomain:app domain:somedomain.uk type:AAAA values:["2a01:4f8:140:903a::2"]
Jan 29 16:11:53 box:dns/manual upsert: app for zone somedomain.uk of type AAAA with values ["2a01:4f8:140:903a::9"]
Jan 29 16:11:53 box:tasks update 22945: {"percent":40,"message":"Downloading image"}
Jan 29 16:11:53 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /var/lib/docker
Jan 29 16:11:53 box:docker downloadImage: docker.domain.uk/cloudron-ocular:v3
Jan 29 16:11:53 box:docker pullImage: will pull docker.domain.uk/cloudron-ocular:v3. auth: no
Jan 29 16:11:53 box:docker Attempt 1 failed. Will retry: Unable to pull image docker.domain.uk/cloudron-ocular:v3. registry error: {"reason":"server error","statusCode":500,"json":null}

NB : auth is still no
Image is visible in browser UI of container registry

Created new container registry app:

• created app password
• updated private registry settings
• desktop terminal : docker logout and login to new registry : success
• desktop terminal : rebuilt app and pushed to new registry (success : checked : visible in app browser UI)
• cloudron install from new registry : failed

Jan 29 16:56:50 box:docker pullImage: will pull mydocker.domain.uk/cloudron-ocular:v5. auth: no
Jan 29 16:56:50 box:docker Attempt 1 failed. Will retry: Unable to pull image mydocker.domain.uk/cloudron-ocular:v5. registry error: {"reason":"server error","statusCode":500,"json":null}

I’m lost.
Did I miss something ?

Superficially, cloudron is failing to access the container registry installed in the cloudron instance. Other connections seem to be working.
So superficially, it looks like a cloudron problem.
But too much grey hair to accept that at face value.

[timconsidine]

Suddden random thought : is there an IPV6 dimension to this ?
No reason for it to affect things (but who knows)

[girish]

@timconsidine I think the issue is that the auth info is not getting picked up for some reason. Do you have https or something in your serverAddress?

Easiest way is to put a console.log here - https://git.cloudron.io/platform/box/-/blob/master/src/docker.js?ref_type=heads#L122

console.log(registryConfig.serverAddress, parsedRef.registry)

Then systemctl restart box and install again. the line above should tell us why they are not matching.