SOLVED Cannot install apps from docker-registry because authentication fails
-
Does anyone have the docker-registry working with authentication? I've tried and it works fine without auth (like my old setup with my custom registry solution), but as soon as I enable user management, the images can't be pushed to the target-cloudron.
Recap of what I was doing:
- installed docker and Cloudron cli on a new linux machine
- installed the docker-registry app on Cloudron X (docker.example.com), added a user ("docker") on X and set its credentials in Cloudron Y settings
- git cloned an app of mine on the linux machine
cloudron login
(on Cloudron Y) anddocker login docker.example.com
docker build -t docker.example.com/my-app .
docker push docker.example.com/my-app
cloudron install --image docker.example.com/my-app -l myapp
Up until the last step everything works fine, but the containers can't get pushed/downloaded on Cloudron Y, this is what happens:
CLI response:
App installation error: Installation failed: Unable to pull image docker.example/my-app. Please check the network or if the image needs authentication. statusCode: 500
App log:
Feb 06 22:35:08 box:docker downloadImage docker.example.com/my-app Feb 06 22:35:08 box:docker Downloading image docker.example.com/my-app. attempt: 1 Feb 06 22:35:08 box:docker pullImage: will pull docker.example.com/my-app. auth: yes Feb 06 22:35:15 box:docker Downloading image docker.example.com/my-app. attempt: 2 Feb 06 22:35:15 box:docker pullImage: will pull docker.example.com/my-app. auth: yes Feb 06 22:35:21 box:docker Downloading image docker.example.com/my-app. attempt: 3 Feb 06 22:35:21 box:docker pullImage: will pull docker.example.com/my-app. auth: yes Feb 06 22:35:26 box:docker Downloading image docker.example.com/my-app. attempt: 4 Feb 06 22:35:26 box:docker pullImage: will pull docker.example.com/my-app. auth: yes Feb 06 22:35:31 box:docker Downloading image docker.example.com/my-app. attempt: 5 Feb 06 22:35:31 box:docker pullImage: will pull docker.example.com/my-app. auth: yes Feb 06 22:35:37 box:docker Downloading image docker.example.com/my-app. attempt: 6 Feb 06 22:35:37 box:docker pullImage: will pull docker.example.com/my-app. auth: yes Feb 06 22:35:42 box:docker Downloading image docker.example.com/my-app. attempt: 7 Feb 06 22:35:42 box:docker pullImage: will pull docker.example.com/my-app. auth: yes Feb 06 22:35:47 box:docker Downloading image docker.example.com/my-app. attempt: 8 Feb 06 22:35:47 box:docker pullImage: will pull docker.example.com/my-app. auth: yes Feb 06 22:35:53 box:docker Downloading image docker.example.com/my-app. attempt: 9 Feb 06 22:35:53 box:docker pullImage: will pull docker.example.com/my-app. auth: yes Feb 06 22:35:58 box:docker Downloading image docker.example.com/my-app. attempt: 10 Feb 06 22:35:58 box:docker pullImage: will pull docker.example.com/my-app. auth: yes Feb 06 22:35:58 box:apptask myapp.cloudrony.com error installing app: BoxError: Unable to pull image docker.example.com/my-app. Please check the network or if the image needs authentication. statusCode: 500 Feb 06 22:35:58 box:apptask myapp.cloudrony.com updating app with values: {"installationState":"error","error":{"message":"Unable to pull image docker.example.com/my-app. Please check the network or if the image needs authentication. statusCode: 500","reason":"Docker Error","taskId":"6145","installationState":"pending_install"}} Feb 06 22:35:58 box:taskworker Task took 55.403 seconds Feb 06 22:35:58 box:tasks setCompleted - 6145: {"result":null,"error":{"stack":"BoxError: Unable to pull image docker.example.com/my-app. Please check the network or if the image needs authentication. statusCode: 500\n at /home/yellowtent/box/src/docker.js:141:40\n at /home/yellowtent/box/node_modules/dockerode/lib/docker.js:119:7\n at /home/yellowtent/box/node_modules/docker-modem/lib/modem.js:265:7\n at IncomingMessage.<anonymous> (/home/yellowtent/box/node_modules/docker-modem/lib/modem.js:284:9)\n at IncomingMessage.emit (events.js:203:15)\n at endReadableNT (_stream_readable.js:1145:12)\n at process._tickCallback (internal/process/next_tick.js:63:19)","name":"BoxError","reason":"Docker Error","details":{},"message":"Unable to pull image docker.example.com/my-app. Please check the network or if the image needs authentication. statusCode: 500"}} Feb 06 22:35:58 box:tasks 6145: {"percent":100,"result":null,"error":{"stack":"BoxError: Unable to pull image docker.example.com/my-app. Please check the network or if the image needs authentication. statusCode: 500\n at /home/yellowtent/box/src/docker.js:141:40\n at /home/yellowtent/box/node_modules/dockerode/lib/docker.js:119:7\n at /home/yellowtent/box/node_modules/docker-modem/lib/modem.js:265:7\n at IncomingMessage.<anonymous> (/home/yellowtent/box/node_modules/docker-modem/lib/modem.js:284:9)\n at IncomingMessage.emit (events.js:203:15)\n at endReadableNT (_stream_readable.js:1145:12)\n at process._tickCallback (internal/process/next_tick.js:63:19)","name":"BoxError","reason":"Docker Error","details":{},"message":"Unable to pull image docker.example.com/my-app. Please check the network or if the image needs authentication. statusCode: 500"}}
Docker logs:
Feb 06 22:35:10 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:10.977369199Z" level=info msg="Attempting next endpoint for pull after error: invalid character '<' looking for beginning of value" Feb 06 22:35:10 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:10.977424454Z" level=error msg="Handler for POST /images/create returned error: invalid character '<' looking for beginning of value" Feb 06 22:35:13 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:13.063771584Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete" Feb 06 22:35:16 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:16.277251730Z" level=info msg="Attempting next endpoint for pull after error: invalid character '<' looking for beginning of value" Feb 06 22:35:16 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:16.277311467Z" level=error msg="Handler for POST /images/create returned error: invalid character '<' looking for beginning of value" Feb 06 22:35:21 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:21.573447894Z" level=info msg="Attempting next endpoint for pull after error: invalid character '<' looking for beginning of value" Feb 06 22:35:21 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:21.573505543Z" level=error msg="Handler for POST /images/create returned error: invalid character '<' looking for beginning of value" Feb 06 22:35:26 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:26.857404427Z" level=info msg="Attempting next endpoint for pull after error: invalid character '<' looking for beginning of value" Feb 06 22:35:26 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:26.857459284Z" level=error msg="Handler for POST /images/create returned error: invalid character '<' looking for beginning of value" Feb 06 22:35:32 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:32.156592002Z" level=info msg="Attempting next endpoint for pull after error: invalid character '<' looking for beginning of value" Feb 06 22:35:32 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:32.156667957Z" level=error msg="Handler for POST /images/create returned error: invalid character '<' looking for beginning of value" Feb 06 22:35:37 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:37.455109662Z" level=info msg="Attempting next endpoint for pull after error: invalid character '<' looking for beginning of value" Feb 06 22:35:37 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:37.455163318Z" level=error msg="Handler for POST /images/create returned error: invalid character '<' looking for beginning of value" Feb 06 22:35:41 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:41.812898340Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete" Feb 06 22:35:42 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:42.576830368Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete" Feb 06 22:35:42 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:42.748936700Z" level=info msg="Attempting next endpoint for pull after error: invalid character '<' looking for beginning of value" Feb 06 22:35:42 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:42.748989045Z" level=error msg="Handler for POST /images/create returned error: invalid character '<' looking for beginning of value" Feb 06 22:35:48 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:48.041591964Z" level=info msg="Attempting next endpoint for pull after error: invalid character '<' looking for beginning of value" Feb 06 22:35:48 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:48.041658716Z" level=error msg="Handler for POST /images/create returned error: invalid character '<' looking for beginning of value" Feb 06 22:35:53 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:53.335916491Z" level=info msg="Attempting next endpoint for pull after error: invalid character '<' looking for beginning of value" Feb 06 22:35:53 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:53.335978423Z" level=error msg="Handler for POST /images/create returned error: invalid character '<' looking for beginning of value" Feb 06 22:35:58 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:58.616154482Z" level=info msg="Attempting next endpoint for pull after error: invalid character '<' looking for beginning of value" Feb 06 22:35:58 my.cloudrony.com dockerd[1433]: time="2021-02-06T21:35:58.616210521Z" level=error msg="Handler for POST /images/create returned error: invalid character '<' looking for beginning of value"
Does anyone have this working as a standalone registry (without gitlab)? No idea what to make of the docker logs with the character thing.
Cheers, M
-
I got also issues with this.
Some auth problem. -
@msbt Thanks for the details report. Indeed, I can confirm that Cloudron Y is unable to pull down a private iamge. I get the same error message.
@BrutalBirdie Is your issue with pushing an image or when downloading the image ? (atleast from the messages on chat the error seemed something else)
-
@msbt OK, so if do
docker login the-x-registry
anddocker pull private-image
it works. So, this looks like some Cloudron bug. -
Looks to be something with the node module (dockerode) we use. It seems to work fine with docker.io private registry just not this custom registry app we have. Have to debug tomorrow.
-
@msbt Found the problem after much debugging. It seems there is some special code that changes the UA string depending on the client. So, the proxyAuth addon does not allow dockerode to authenticate properly.
https://git.cloudron.io/cloudron/box/-/commit/1d0ad3cb47f85b05eabb31853c8c3a585d06c2e9 is the fix. It's really just changing
docker-client
todocker
. If you apply the patch, restart the box code and also go to Docker registry app -> Location -> Save to regenerate the nginx config. -
@girish nice, great find! I'll check it out tomorrow
-
looking good @girish, just made those changes and did a cloudron install, worked like a charm!