Recent app updates error

[Lanhild]

Lately, when the app update process tries to pull the application image, it often results in an error.

It's the not the first time I've seen this happen on Cloudron, but it's happened a lot more than before recently.

[girish]

@Lanhild after reading the rate limit docs many times, it seems maybe that pull limits are rate limited to 10 per hour for unauthenticated users. we have started adding a fallback for dockerhub and all the latest images are now pushed into quay.io as well - https://quay.io/organization/cloudron . Next release, will automatically try to pull from quay, when dockerhub rate limits are pushed (maybe we will switch to quay entirely at some point, have to see).

For your immediate problem, in /home/yellowtent/box/src/docker.js , can you comment out this one line below? Then, put your credentials into the Private Registry Config in the Settings page. I think that should sort it out temporarily. For the next release, this won't be needed.

async function getAuthConfig(image) {
    // https://github.com/docker/distribution/blob/release/2.7/reference/normalize.go#L62
    const parts = image.split('/');
    // if (parts.length === 1 || (parts[0].match(/[.:]/) === null)) return null; // public docker registry COMMENT THIS OUT

[Lanhild]

Another example

[girish]

The metabase image is here - https://hub.docker.com/layers/cloudron/com.metabase.cloudronapp/202412100020330000/images/sha256-1b4bb3178971ddcbc2ae336e5d21e22d4df1afac1e2dc977df04f6042e7633b0?context=explore

Are you able to run docker pull cloudron/com.metabase.cloudronapp:202412100020330000 manually on the server? But it returned a 500, seems like some docker hub issue in your region? The pull works fine for me here.

[Lanhild]

@girish Yes, I can manually pull it. It seems like the issue is momentaneous and only happens rarely. Though, I noticed it only happens when applications update automatically.

[Lanhild]

@girish I have found the issue:

sudo docker pull cloudron/org.apache.guacamole.cloudronapp:202412061725490000
[sudo] password for sysadmin: 
Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

[Lanhild]

@Lanhild That's an issue for any Cloudron server that have multiple copies of the same app installed. Any app with automatic updates activated will face this issue. That's pretty critical.

[girish]

@Lanhild are you also using that IP address (maybe via your home network) for other development/hosting ? Are you using Private docker registry against dockerhub with your credentials ?

Cloudron pays for DockerHub , but reading https://www.docker.com/pricing/ . https://docs.docker.com/docker-hub/download-rate-limit/ has a wide range of limits. It's all very complicated ...

I read a suggestion somewhere that you can "docker login" with your credentials on the server and this helps the rate limit.

[Lanhild]

@girish said in Recent app updates error:

@Lanhild are you also using that IP address (maybe via your home network) for other development/hosting ? Are you using Private docker registry against dockerhub with your credentials ?

I am not.

Cloudron pays for DockerHub , but reading https://www.docker.com/pricing/ . https://docs.docker.com/docker-hub/download-rate-limit/ has a wide range of limits. It's all very complicated ...

Indeed, I had foudn this documentation page, but it is very confusing to read.

I read a suggestion somewhere that you can "docker login" with your credentials on the server and this helps the rate limit.

I'll try this and report.

[Lanhild]

@Lanhild Looks like this solved it. I'll open a new topic if this were to happen again at the same volume.

[Lanhild]

@girish I've docker login'ed on the root account of my VPS. Pulling an image from the CLI as root works well, but I'm still hitting the same error when installing/updating an app from the dashboard. Should I docker login on another user?

[girish]

@Lanhild Ha, we just hit this on one of our servers as well. Investigating what the issue is.

[girish]

@Lanhild after reading the rate limit docs many times, it seems maybe that pull limits are rate limited to 10 per hour for unauthenticated users. we have started adding a fallback for dockerhub and all the latest images are now pushed into quay.io as well - https://quay.io/organization/cloudron . Next release, will automatically try to pull from quay, when dockerhub rate limits are pushed (maybe we will switch to quay entirely at some point, have to see).

For your immediate problem, in /home/yellowtent/box/src/docker.js , can you comment out this one line below? Then, put your credentials into the Private Registry Config in the Settings page. I think that should sort it out temporarily. For the next release, this won't be needed.

async function getAuthConfig(image) {
    // https://github.com/docker/distribution/blob/release/2.7/reference/normalize.go#L62
    const parts = image.split('/');
    // if (parts.length === 1 || (parts[0].match(/[.:]/) === null)) return null; // public docker registry COMMENT THIS OUT

[girish]

@Lanhild you can also run this:

TOKEN=$(curl "https://auth.docker.io/token?service=registry.docker.io&scope=repository:ratelimitpreview/test:pull" | jq -r .token)
curl --head -H "Authorization: Bearer $TOKEN" https://registry-1.docker.io/v2/ratelimitpreview/test/manifests/latest

this one finds your ratelimit . from what we have seen, this happens only with servers that have IPv6. It looks like docker's ipv6 rate limiting is not very good. you might have to do apt install jq for the above to work.

[girish]

https://github.com/ubicloud/ubicloud/discussions/2244#discussioncomment-11232417 and https://www.docker.com/blog/beta-ipv6-support-on-docker-hub-registry/ says "the first 64 bits of IPv6 addresses" for ratelimiting

[girish]

On a brand new DO Ubuntu server, the first pull fails.

root@my:~# TOKEN=$(curl "https://auth.docker.io/token?service=registry.docker.io&scope=repository:ratelimitpreview/test:pull" | jq -r .token)
curl --head -H "Authorization: Bearer $TOKEN" https://registry-1.docker.io/v2/ratelimitpreview/test/manifests/latest
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  5429    0  5429    0     0  17471      0 --:--:-- --:--:-- --:--:-- 17512
HTTP/1.1 200 OK
content-length: 527
content-type: application/vnd.docker.distribution.manifest.v2+json
docker-content-digest: sha256:c2d41d2ba6d8b7b4a3ffec621578eb4d9a0909df29dfa2f6fd8a2e5fd0836aed
docker-distribution-api-version: registry/2.0
etag: "sha256:c2d41d2ba6d8b7b4a3ffec621578eb4d9a0909df29dfa2f6fd8a2e5fd0836aed"
date: Sat, 14 Dec 2024 15:41:35 GMT
strict-transport-security: max-age=31536000
ratelimit-limit: 100;w=21600
ratelimit-remaining: 0;w=21600
docker-ratelimit-source: 2604:a880:2:d1::