Docker Error 500 - Unable to pull image on same instance

[nottheend]

I am encountering this error for my selfhosted private Docker registry, how to come closer to the probable authentication error?

Docker Error - Unable to pull image mydomain.com/myusername/myapp:1.0.0 Please check the network or if the image needs authentication. statusCode 500

In the logs are 10 not successful pull attempts visible.

The Docker registry is hosted on the same Cloudron server, it is Docker Registry App from official App Store.

In the settings if my Cloudron instance, I configured a Private Docker Registry with the same Credentials I used to successfully push the Docker build from local. I changed from Docker to Other just to be sure, with the same credentials.
After building the build, I can successfullypush it with the very same credentials to the private Docker Hub

[nebulon]

Interesting use-case to have the registry on the same server. For a start can you check the logs of the registry app? Also check if you can curl or otherwise run a successful request from your host system to the registry itself.

[nottheend]

Thank you!
Basically, pushing a image works to the registry with docker push also on the server (see below.

But I seem to have change something and while deploying the app with the cloudron npm command, I get an error of an invalid manifest file. Will have to dig into it (I have all required parameters). Probably my project structure is wrong:

App installation error: Installation failed: Unable to pull image dockerhub.domain.com/myuser/myapp. message: (HTTP code 404) unexpected - manifest for dockerhub.domain.com/myuser/myapp:latest not found: manifest unknown: manifest unknown statusCode: 404

Logs on server for docker push:
2024-08-11T13:35:40.000Z time="2024-08-11T13:35:40.047968853Z" level=info msg="response completed" go.version=go1.20.8 http.request.host="localhost:5000" http.request.id=ea838230-bda2-414b-aa83-599c00a48956 http.request.method=GET http.request.remoteaddr="[::1]:59814" http.request.uri="/v2" http.request.useragent="Mozilla (CloudronHealth)" http.response.contenttype="text/html; charset=utf-8" http.response.duration="104.432µs" http.response.status=301 http.response.written=39
2.

Below is the response for curl -v ```
https://dockerhub.domain.com/v2/_catalog

[user]$ curl -v https://dockerhub.domain.com/v2/_catalog

• Host dockerhub.domain.com:443 was resolved.
• IPv6: (none)
• IPv4: xxx.xxx.xxx.xxx
• Trying xxx.xxx.xxx.xxx:443...
• Connected to dockerhub.domain.com (xxx.xxx.xxx.xxx) port 443
• ALPN: curl offers h2,http/1.1
• TLSv1.3 (OUT), TLS handshake, Client hello (1):
• CAfile: /etc/pki/tls/certs/ca-bundle.crt
• CApath: none
• TLSv1.3 (IN), TLS handshake, Server hello (2):
• TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
• TLSv1.3 (IN), TLS handshake, Certificate (11):
• TLSv1.3 (IN), TLS handshake, CERT verify (15):
• TLSv1.3 (IN), TLS handshake, Finished (20):
• TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
• TLSv1.3 (OUT), TLS handshake, Finished (20):
• SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / x25519 / id-ecPublicKey
• ALPN: server accepted h2
• Server certificate:
• subject: CN=*.domain.com
• start date: 2024 GMT
• expire date: 2024 GMT
• subjectAltName: host "dockerhub.domain.com" matched cert's "*.domain.com"
• issuer: C=US; O=Let's Encrypt; CN=E5
• SSL certificate verify ok.
• Certificate level 0: Public key type EC/secp384r1 (384/192 Bits/secBits), signed using ecdsa-with-SHA384
• Certificate level 1: Public key type EC/secp384r1 (384/192 Bits/secBits), signed using sha256WithRSAEncryption
• Certificate level 2: Public key type RSA (4096/152 Bits/secBits), signed using sha256WithRSAEncryption
• using HTTP/2
• [HTTP/2] [1] OPENED stream for https://dockerhub.domain.com/v2/_catalog
• [HTTP/2] [1] [:method: GET]
• [HTTP/2] [1] [:scheme: https]
• [HTTP/2] [1] [:authority: dockerhub.domain.com]
• [HTTP/2] [1] [:path: /v2/_catalog]
• [HTTP/2] [1] [user-agent: curl/8.6.0]
• [HTTP/2] [1] [accept: /]

GET /v2/_catalog HTTP/2
Host: dockerhub.domain.com
User-Agent: curl/8.6.0
Accept: /

• TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
• TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
• old SSL session ID is stale, removing
  < HTTP/2 302
  < server: nginx
  < date: Sun, 11 Aug 2024 13:19:02 GMT
  < content-type: text/html
  < content-length: 138
  < location: https://dockerhub.domain.com/login?redirect=/v2/_catalog
  < strict-transport-security: max-age=63072000
  < x-xss-protection: 1; mode=block
  < x-download-options: noopen
  < x-content-type-options: nosniff
  < x-permitted-cross-domain-policies: none
  < referrer-policy: same-origin
  < cache-control: no-cache
  < set-cookie: authToken=; Path=/; Expires=Thu, 01 Jan 1970 00:00:00 GMT
  <
  <html>
  <head><title>302 Found</title></head>
  <body>
  <center><h1>302 Found</h1></center>
  <hr><center>nginx</center>
  </body>
  </html>
• Connection #0 to host dockerhub.domain.com left intact

[nebulon]

@nottheend said in Docker Error 500 - Unable to pull image on same instance:

App installation error: Installation failed: Unable to pull image dockerhub.domain.com/myuser/myapp. message: (HTTP code 404) unexpected - manifest for dockerhub.domain.com/myuser/myapp:latest not found: manifest unknown: manifest unknown statusCode: 404

I am a bit lost by now. Could it just be that the latest tag and or repository simply does not exist at dockerhub.domain.com/myuser/myapp ?

Can you otherwise docker pull from for example your laptop with the same image URI ?

[nottheend]

The error doesn't appear anymore. Not sure why (cloudron updated to v8 meanwhile).

The errors remaining are "typical build image errors". Thanks for your support!

[timconsidine]

Hmmm
I am now getting this error also

App is being installed.
=> Queued .
=> Registering subdomains
=> Downloading image ........................................
App installation error: Installation failed: Unable to pull image docker.domain.uk/cloudron-ocular:v1. registry error: {"reason":"server error","statusCode":500,"json":null}

But I can see the image in my docker registry so docker build and docker push is working.
The docker registry is on the Cloudron instance, but this has never been a problem before.

Have ensured docker login and cloudron install etc work

Running Cloudron v8.2.3 on Ubuntu 22.04

App install logs show :

box:docker pullImage: will pull docker.domain.uk/cloudron-ocular:v1. auth: no

Is that auth: no significant ?

Final log entry :

box:apptask run: app error for state pending_install: BoxError: Unable to pull image docker.domain.uk/cloudron-ocular:v1. registry error: {"reason":"server error","statusCode":500,"json":null} at pullImage (/home/yellowtent/box/src/docker.js:150:50) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async /home/yellowtent/box/src/docker.js:195:122 at async promiseRetry (/home/yellowtent/box/src/promise-retry.js:17:20) at async Object.downloadImage (/home/yellowtent/box/src/docker.js:194:5) at async downloadImage (/home/yellowtent/box/src/apptask.js:243:5) at async install (/home/yellowtent/box/src/apptask.js:337:5) { reason: 'Docker Error', details: {} }

box:tasks setCompleted - 22828: {"result":null,"error":{"stack":"BoxError: Unable to pull image docker.domain.uk/cloudron-ocular:v1. registry error: {"reason":"server error","statusCode":500,"json":null}\n at pullImage (/home/yellowtent/box/src/docker.js:150:50)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async /home/yellowtent/box/src/docker.js:195:122\n at async promiseRetry (/home/yellowtent/box/src/promise-retry.js:17:20)\n at async Object.downloadImage (/home/yellowtent/box/src/docker.js:194:5)\n at async downloadImage (/home/yellowtent/box/src/apptask.js:243:5)\n at async install (/home/yellowtent/box/src/apptask.js:337:5)","name":"BoxError","reason":"Docker Error","details":{},"message":"Unable to pull image docker.domain.uk/cloudron-ocular:v1. registry error: {"reason":"server error","statusCode":500,"json":null}"}}

box:tasks update 22828: {"percent":100,"result":null,"error":{"stack":"BoxError: Unable to pull image docker.domain.uk/cloudron-ocular:v1. registry error: {"reason":"server error","statusCode":500,"json":null}\n at pullImage (/home/yellowtent/box/src/docker.js:150:50)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async /home/yellowtent/box/src/docker.js:195:122\n at async promiseRetry (/home/yellowtent/box/src/promise-retry.js:17:20)\n at async Object.downloadImage (/home/yellowtent/box/src/docker.js:194:5)\n at async downloadImage (/home/yellowtent/box/src/apptask.js:243:5)\n at async install (/home/yellowtent/box/src/apptask.js:337:5)","name":"BoxError","reason":"Docker Error","details":{},"message":"Unable to pull image docker.domain.uk/cloudron-ocular:v1. registry error: {"reason":"server error","statusCode":500,"json":null}"}}

[nebulon]

Given that you see 500 status errors from your registry, can you check the logs there? 500 if used correctly refer to remote server errors. Auth failure would be in the 400 range (403, 402, ..)

[timconsidine]

In the docker registry app, logs show endless lines like this, but I don’t see errors as such :

Jan 23 11:41:20 127.0.0.1 - - [23/Jan/2025:11:41:20 +0000] "GET /v2 HTTP/1.0" 301 39 "" "Mozilla (CloudronHealth)"

Jan 23 11:41:20 time="2025-01-23T11:41:20.113222934Z" level=info msg="response completed" go.version=go1.20.8 http.request.host="localhost:5000" http.request.id=216d4214-238a-409b-930b-0a1ba5bf2f02 http.request.method=GET http.request.remoteaddr="127.0.0.1:56446" http.request.uri="/v2" http.request.useragent="Mozilla (CloudronHealth)" http.response.contenttype="text/html; charset=utf-8" http.response.duration="258.188µs" http.response.status=301 http.response.written=39

Jan 23 11:41:30 ::1 - - [23/Jan/2025:11:41:30 +0000] "GET /v2 HTTP/1.0" 301 39 "" "Mozilla (CloudronHealth)"

Jan 23 11:41:30 time="2025-01-23T11:41:30.129205782Z" level=info msg="response completed" go.version=go1.20.8 http.request.host="localhost:5000" http.request.id=ea05e778-e39b-47cd-8e12-5bb881512b3b http.request.method=GET http.request.remoteaddr="[::1]:54150" http.request.uri="/v2" http.request.useragent="Mozilla (CloudronHealth)" http.response.contenttype="text/html; charset=utf-8" http.response.duration="82.643µs" http.response.status=301 http.response.written=39

Jan 23 11:41:30 172.18.0.1 - - [23/Jan/2025:11:41:30 +0000] "GET /v2 HTTP/1.1" 301 39 "-" "Mozilla (CloudronHealth)”

Restarted the docker registry app on Cloudron, but same behaviour.
Image is visible in the registry app UI.

On local device I can do docker pull docker.domain.uk/image:tag so I guess the docker registry is ok, UI and images in app visible. And of course the docker push command worked.

[girish]

@timconsidine said in Docker Error 500 - Unable to pull image on same instance:

Is that auth: no significant ?

if the registry requires authentication then yes, this appears significant. Can you check the settings in Private Registry? Are you also able to docker pull on the cloudron itself?

[timconsidine]

@girish yes I can docker push and docker pull on my desktop so auth here is not a problem.
Also I have logged out and logged back in.
Cloudron CLI is able to create the app and sub-domain, so my cloudron auth is ok.
And I can do cloudron uninstall —app blah

[girish]

@timconsidine the cloudron cli auth is different from the registry auth . I guess you had done docker login at some point in your desktop and this is why push/pull works.

The code is supposed to get the authentication info from the Settings -> Private Docker Registry in this function - https://git.cloudron.io/platform/box/-/blob/master/src/docker.js?ref_type=heads#L112 . Have you configured Private Docker Registry? If so, maybe you can put a couple of console.log in that function to see what is going wrong. Maybe something wrong in the parsing logic

[timconsidine]

Thanks @girish
In a shipping container in DE, will have to investigate once I get back
Not least because I don’t understand about settings but will dive in