AppStore returns 424

[osobo]

Dear @girish it seems the App store is down again since yesterday
Thank you for your consideration and awaiting the repair.
Regards

[necrevistonnezr]

It's working for me?

[nebulon]

@osobo also working here, do you see any errors or blocked network requests in your browser dev tools?

[osobo]

Hello @nebulon
Yes I've the following error 424
I've tried with a VPN but no change...
Thanks for your support !

[threetrees3]

Getting similar issues with appstore today, console errors;

Running dashboard version  7.7.2 index.js:786
XHRGET
https://domain.net/api/v1/appstore/apps
[HTTP/2 424  30115ms]

Failed to get apps. Will retry. 
Object { name: "ClientError", statusCode: 424, status: "Failed Dependency", message: "Timeout of 30000ms exceeded" }
appstore.js:780
XHRGET
https://domain.net/api/v1/appstore/apps
[HTTP/2 424  30101ms]

Failed to get apps. Will retry. 
Object { name: "ClientError", statusCode: 424, status: "Failed Dependency", message: "Timeout of 30000ms exceeded" }
appstore.js:780

[girish]

@osobo @threetrees3 does https://my.demo.cloudron.io/#/appstore work for you?

Can you go to Services -> Unbound -> Restart and try after that?

[osobo]

@girish said in AppStore returns 424:

@osobo @threetrees3 does https://my.demo.cloudron.io/#/appstore work for you?

Yes it works for me

@girish said in AppStore returns 424:

Can you go to Services -> Unbound -> Restart and try after that?

I've done it, but it doesn't solve the problem.

Thanks for your support !

[girish]

@osobo Can you SSH into your server and check if curl https://api.cloudron.io/api/v1/apps works ? It should dump a lot of json output (the app listing).

[osobo]

@girish said in AppStore returns 424:

check if curl https://api.cloudron.io/api/v1/apps works

Yes it works. Very long dump indeed. Do you need it ?

[threetrees3]

Just a few hours ago it was working and now weirdly it's doing the same sort of error again.

curl https://api.cloudron.io/api/v1/apps lists all the apps as expected: https://privatebin.io/?939f91ba3497831d#BMQW6u6XzjACJTj62U38F64G2LyGRu6XGEEz7tMPUUaf

[nebulon]

So if your Cloudron can query the appstore reliably (api.cloudron.io) but the calls directly to your Cloudron time out, then maybe your server has a hard time to keep answering with the 30s timeout limit. How is your server load during the times it fails?

[threetrees3]

Server's load appears fine; this is top when the app store fails to load - no steal and load averages less than 1.

Edited to add:
Some context. No issue until the last two days.
This is a VPS on a Hosthatch VPS hosted in Singapore. 16G, 3vCPU.
Not behind Cloudflare.

Edit2:
Trial restarting Unbound -> same error
Trial reboot -> Same error
Trial multiple browsers including mobile browser -> appstore still doesn't load.

[nebulon]

Can you also check the logs at /home/yellowtent/platformdata/logs/box.log on the server when you try to open the appstore in the dashboard?

[girish]

@osobo @threetrees3 if you can contact us at support@cloudron.io , we can debug further .

[threetrees3]

@girish

Have done so.

tail -f /home/yellowtent/platformdata/logs/box.log
2024-04-22T14:00:50.182Z box:apphealthmonitor app health: 13 running / 0 stopped / 0 unresponsive
2024-04-22T14:01:00.152Z box:apphealthmonitor app health: 13 running / 0 stopped / 0 unresponsive
2024-04-22T14:01:10.140Z box:apphealthmonitor app health: 13 running / 0 stopped / 0 unresponsive
2024-04-22T14:01:20.150Z box:apphealthmonitor app health: 13 running / 0 stopped / 0 unresponsive
2024-04-22T14:01:30.152Z box:apphealthmonitor app health: 13 running / 0 stopped / 0 unresponsive
2024-04-22T14:01:40.159Z box:apphealthmonitor app health: 13 running / 0 stopped / 0 unresponsive
2024-04-22T14:01:50.154Z box:apphealthmonitor app health: 13 running / 0 stopped / 0 unresponsive
2024-04-22T14:02:00.152Z box:apphealthmonitor app health: 13 running / 0 stopped / 0 unresponsive
2024-04-22T14:02:10.162Z box:apphealthmonitor app health: 13 running / 0 stopped / 0 unresponsive
2024-04-22T14:02:20.158Z box:apphealthmonitor app health: 13 running / 0 stopped / 0 unresponsive
2024-04-22T14:02:30.152Z box:apphealthmonitor app health: 13 running / 0 stopped / 0 unresponsive
2024-04-22T14:02:40.132Z box:apphealthmonitor app health: 13 running / 0 stopped / 0 unresponsive
2024-04-22T14:02:50.161Z box:apphealthmonitor app health: 13 running / 0 stopped / 0 unresponsive
2024-04-22T14:03:00.156Z box:apphealthmonitor app health: 13 running / 0 stopped / 0 unresponsive
2024-04-22T14:03:10.163Z box:apphealthmonitor app health: 13 running / 0 stopped / 0 unresponsive
2024-04-22T14:03:20.148Z box:apphealthmonitor app health: 13 running / 0 stopped / 0 unresponsive
2024-04-22T14:03:30.172Z box:apphealthmonitor app health: 13 running / 0 stopped / 0 unresponsive
2024-04-22T14:03:40.148Z box:apphealthmonitor app health: 13 running / 0 stopped / 0 unresponsive

[osobo]

@girish Thank you. The App Store is back today and everything's working fine !
Strangely the logs are however similar to @threetrees3's ones
Thanks anyway for the support and the fix !
Best regards

tail -f /home/yellowtent/platformdata/logs/box.log
2024-04-23T01:26:10.157Z box:apphealthmonitor app health: 7 running / 0 stopped / 0 unresponsive
2024-04-23T01:26:20.181Z box:apphealthmonitor app health: 7 running / 0 stopped / 0 unresponsive
2024-04-23T01:26:30.177Z box:apphealthmonitor app health: 7 running / 0 stopped / 0 unresponsive
2024-04-23T01:26:40.146Z box:apphealthmonitor app health: 7 running / 0 stopped / 0 unresponsive
2024-04-23T01:26:50.152Z box:apphealthmonitor app health: 7 running / 0 stopped / 0 unresponsive
2024-04-23T01:27:00.164Z box:apphealthmonitor app health: 7 running / 0 stopped / 0 unresponsive
2024-04-23T01:27:10.145Z box:apphealthmonitor app health: 7 running / 0 stopped / 0 unresponsive
2024-04-23T01:27:20.165Z box:apphealthmonitor app health: 7 running / 0 stopped / 0 unresponsive
2024-04-23T01:27:30.177Z box:apphealthmonitor app health: 7 running / 0 stopped / 0 unresponsive
2024-04-23T01:27:40.154Z box:apphealthmonitor app health: 7 running / 0 stopped / 0 unresponsive
2024-04-23T01:27:50.141Z box:apphealthmonitor app health: 7 running / 0 stopped / 0 unresponsive

[girish]

@osobo I am debugging this issue further as @threetrees3 has given me access. I managed to reproduce the problem . It seems that superagent throws some error as superagent: double callback bug sporadically. I am investigaging. Can you check if your box.log has a similar message somewhere ?

[girish]

Got a reproducer.

'use strict';

const superagent = require('superagent'),
    safe = require('safetydance');

(async function () {
    const [error, reponse] = await safe(superagent.get('https://api.cloudron.io/api/v1/apps').timeout(30000).ok(() => true));
    console.log(error);
})();

The above results in:

Error: Timeout of 30000ms exceeded
    at RequestBase._timeoutError (/home/yellowtent/box/node_modules/superagent/lib/request-base.js:712:17)
    at Timeout.<anonymous> (/home/yellowtent/box/node_modules/superagent/lib/request-base.js:727:12)
    at listOnTimeout (node:internal/timers:569:17)
    at process.processTimers (node:internal/timers:512:7) {
  timeout: 30000,
  code: 'ECONNABORTED',
  errno: 'ETIME',
  response: undefined
}
superagent: double callback bug

That it takes > 30 seconds for the response seems to be server related. On my server(s), I get response in 2 seconds.

# time node bug.js 
null

real	0m2.046s
user	0m0.156s
sys	0m0.035s

[girish]

@osobo @threetrees3 I have reported the bug upstream at https://github.com/ladjs/superagent/issues/1801 . In the meantime, I have bumped timeout to 60s in our code as a workaround to not hit that bug.

[girish]

@osobo @threetrees3 are your servers hosted in asia / digitalocean ?

https://status.digitalocean.com/incidents/21gg18q1ddmv "Subsea Fiber Faults Between Europe and Asia" . I think this is the cause of the slow network issues.