AppStore returns 424

[threetrees3]

Just a few hours ago it was working and now weirdly it's doing the same sort of error again.

curl https://api.cloudron.io/api/v1/apps lists all the apps as expected: https://privatebin.io/?939f91ba3497831d#BMQW6u6XzjACJTj62U38F64G2LyGRu6XGEEz7tMPUUaf

[nebulon]

So if your Cloudron can query the appstore reliably (api.cloudron.io) but the calls directly to your Cloudron time out, then maybe your server has a hard time to keep answering with the 30s timeout limit. How is your server load during the times it fails?

[threetrees3]

Server's load appears fine; this is top when the app store fails to load - no steal and load averages less than 1.

Edited to add:
Some context. No issue until the last two days.
This is a VPS on a Hosthatch VPS hosted in Singapore. 16G, 3vCPU.
Not behind Cloudflare.

Edit2:
Trial restarting Unbound -> same error
Trial reboot -> Same error
Trial multiple browsers including mobile browser -> appstore still doesn't load.

[nebulon]

Can you also check the logs at /home/yellowtent/platformdata/logs/box.log on the server when you try to open the appstore in the dashboard?

[girish]

@osobo @threetrees3 if you can contact us at support@cloudron.io , we can debug further .

[threetrees3]

@girish

Have done so.

tail -f /home/yellowtent/platformdata/logs/box.log
2024-04-22T14:00:50.182Z box:apphealthmonitor app health: 13 running / 0 stopped / 0 unresponsive
2024-04-22T14:01:00.152Z box:apphealthmonitor app health: 13 running / 0 stopped / 0 unresponsive
2024-04-22T14:01:10.140Z box:apphealthmonitor app health: 13 running / 0 stopped / 0 unresponsive
2024-04-22T14:01:20.150Z box:apphealthmonitor app health: 13 running / 0 stopped / 0 unresponsive
2024-04-22T14:01:30.152Z box:apphealthmonitor app health: 13 running / 0 stopped / 0 unresponsive
2024-04-22T14:01:40.159Z box:apphealthmonitor app health: 13 running / 0 stopped / 0 unresponsive
2024-04-22T14:01:50.154Z box:apphealthmonitor app health: 13 running / 0 stopped / 0 unresponsive
2024-04-22T14:02:00.152Z box:apphealthmonitor app health: 13 running / 0 stopped / 0 unresponsive
2024-04-22T14:02:10.162Z box:apphealthmonitor app health: 13 running / 0 stopped / 0 unresponsive
2024-04-22T14:02:20.158Z box:apphealthmonitor app health: 13 running / 0 stopped / 0 unresponsive
2024-04-22T14:02:30.152Z box:apphealthmonitor app health: 13 running / 0 stopped / 0 unresponsive
2024-04-22T14:02:40.132Z box:apphealthmonitor app health: 13 running / 0 stopped / 0 unresponsive
2024-04-22T14:02:50.161Z box:apphealthmonitor app health: 13 running / 0 stopped / 0 unresponsive
2024-04-22T14:03:00.156Z box:apphealthmonitor app health: 13 running / 0 stopped / 0 unresponsive
2024-04-22T14:03:10.163Z box:apphealthmonitor app health: 13 running / 0 stopped / 0 unresponsive
2024-04-22T14:03:20.148Z box:apphealthmonitor app health: 13 running / 0 stopped / 0 unresponsive
2024-04-22T14:03:30.172Z box:apphealthmonitor app health: 13 running / 0 stopped / 0 unresponsive
2024-04-22T14:03:40.148Z box:apphealthmonitor app health: 13 running / 0 stopped / 0 unresponsive

[osobo]

@girish Thank you. The App Store is back today and everything's working fine !
Strangely the logs are however similar to @threetrees3's ones
Thanks anyway for the support and the fix !
Best regards

tail -f /home/yellowtent/platformdata/logs/box.log
2024-04-23T01:26:10.157Z box:apphealthmonitor app health: 7 running / 0 stopped / 0 unresponsive
2024-04-23T01:26:20.181Z box:apphealthmonitor app health: 7 running / 0 stopped / 0 unresponsive
2024-04-23T01:26:30.177Z box:apphealthmonitor app health: 7 running / 0 stopped / 0 unresponsive
2024-04-23T01:26:40.146Z box:apphealthmonitor app health: 7 running / 0 stopped / 0 unresponsive
2024-04-23T01:26:50.152Z box:apphealthmonitor app health: 7 running / 0 stopped / 0 unresponsive
2024-04-23T01:27:00.164Z box:apphealthmonitor app health: 7 running / 0 stopped / 0 unresponsive
2024-04-23T01:27:10.145Z box:apphealthmonitor app health: 7 running / 0 stopped / 0 unresponsive
2024-04-23T01:27:20.165Z box:apphealthmonitor app health: 7 running / 0 stopped / 0 unresponsive
2024-04-23T01:27:30.177Z box:apphealthmonitor app health: 7 running / 0 stopped / 0 unresponsive
2024-04-23T01:27:40.154Z box:apphealthmonitor app health: 7 running / 0 stopped / 0 unresponsive
2024-04-23T01:27:50.141Z box:apphealthmonitor app health: 7 running / 0 stopped / 0 unresponsive

[girish]

@osobo I am debugging this issue further as @threetrees3 has given me access. I managed to reproduce the problem . It seems that superagent throws some error as superagent: double callback bug sporadically. I am investigaging. Can you check if your box.log has a similar message somewhere ?

[girish]

Got a reproducer.

'use strict';

const superagent = require('superagent'),
    safe = require('safetydance');

(async function () {
    const [error, reponse] = await safe(superagent.get('https://api.cloudron.io/api/v1/apps').timeout(30000).ok(() => true));
    console.log(error);
})();

The above results in:

Error: Timeout of 30000ms exceeded
    at RequestBase._timeoutError (/home/yellowtent/box/node_modules/superagent/lib/request-base.js:712:17)
    at Timeout.<anonymous> (/home/yellowtent/box/node_modules/superagent/lib/request-base.js:727:12)
    at listOnTimeout (node:internal/timers:569:17)
    at process.processTimers (node:internal/timers:512:7) {
  timeout: 30000,
  code: 'ECONNABORTED',
  errno: 'ETIME',
  response: undefined
}
superagent: double callback bug

That it takes > 30 seconds for the response seems to be server related. On my server(s), I get response in 2 seconds.

# time node bug.js 
null

real	0m2.046s
user	0m0.156s
sys	0m0.035s

[girish]

@osobo @threetrees3 I have reported the bug upstream at https://github.com/ladjs/superagent/issues/1801 . In the meantime, I have bumped timeout to 60s in our code as a workaround to not hit that bug.

[girish]

@osobo @threetrees3 are your servers hosted in asia / digitalocean ?

https://status.digitalocean.com/incidents/21gg18q1ddmv "Subsea Fiber Faults Between Europe and Asia" . I think this is the cause of the slow network issues.

[threetrees3]

@girish

My server is hosted in Singapore by Hosthatch (M247). There does appear to be a significant speed issues with an eclectic selection of hosts.

Disabling IPv6 seems to help a bit. Though, I am migrating servers as we speak.

Thanks again for working out the issue! Much appreciate the speedy support.

---------------------------------------------------------------------------
 Speedtest.net (Region: EUROPE)
---------------------------------------------------------------------------
 Location         Latency     Loss    DL Speed       UP Speed       Server

 ISP: HostHatch LLC

 Nearest          1.15 ms     0.0%    8454.38 Mbps   7303.20 Mbps   MyRepublic Singapore - Singapore

 London, UK       243.81 ms   0.0%    475.24 Mbps    327.99 Mbps    M247 Ltd - London
 Manchester, UK   293.50 ms   N/A     273.58 Mbps    245.72 Mbps    Vodafone UK - Manchester
 Dublin, IE       333.20 ms   0.0%    1162.39 Mbps   229.84 Mbps    Three Ireland - Dublin
 Amsterdam, NL    343.10 ms   0.0%    1.15 Mbps      229.11 Mbps    Melbicom - Amsterdam
 Eygelshoen, NL   336.44 ms   0.0%    1.04 Mbps      90.83 Mbps     SkyLink Data Center BV - Eygelshoven
 Paris, FR        309.83 ms   N/A     883.20 Mbps    261.13 Mbps    ORANGE FRANCE - Paris
 Marseille, FR    307.23 ms   0.0%    2459.84 Mbps   252.28 Mbps    GSL Networks - Marseille
 Madrid, ES       253.34 ms   0.0%    1220.94 Mbps   272.96 Mbps    Orange - Madrid
 Barcelona, ES    320.58 ms   0.0%    1059.18 Mbps   248.52 Mbps    Adamo - Barcelona
 Lisbon, PT       352.03 ms   0.0%    2.70 Mbps      204.72 Mbps    Edgoo Networks - Lisbon
 Rome, IT         328.83 ms   0.0%    2288.38 Mbps   225.60 Mbps    TIM SpA - Rome
 Milan, IT        310.19 ms   0.0%    1389.43 Mbps   258.39 Mbps    Fastweb SpA - Milan
 Zurich, CH       345.50 ms   0.0%    2.30 Mbps      214.52 Mbps    Sunrise Communication AG - Zurich
 Frankfurt, DE    311.55 ms   0.0%    1550.73 Mbps   252.18 Mbps    Deutsche Telekom - Frankfurt am Main
 Berlin, DE       350.03 ms   N/A     1.16 Mbps      109.76 Mbps    Misaka Network, Inc. - Berlin
 Vienna, AT       314.64 ms   0.0%    2455.41 Mbps   251.89 Mbps    A1 Telekom Austria AG - Vienna
 Budapest, HU     362.27 ms   0.0%    0.56 Mbps      199.73 Mbps    ATW Internet Kft. - Budapest
 Gdansk, PL       322.86 ms   0.0%    2236.89 Mbps   251.03 Mbps    T-Mobile Polska S.A. - Gdańsk
 Warsaw, PL       322.19 ms   0.0%    2035.88 Mbps   244.78 Mbps    Orange Polska S.A. - Warsaw
 Lviv, UA         370.95 ms   0.0%    5.47 Mbps      204.86 Mbps    Kyivstar - Lviv
 Kyiv, UA         370.82 ms   0.0%    7.30 Mbps      224.98 Mbps    O3 - Kyiv
 Bucharest, RO    393.82 ms   0.0%    2.77 Mbps      220.72 Mbps    Orange Romania SA - Bucuresti
 Timisoara, RO    352.97 ms   0.0%    0.69 Mbps      201.51 Mbps    Digi
 Helsinki, FI     354.19 ms   0.0%    2.16 Mbps      194.89 Mbps    Elisa Oyj - Helsinki
 Stockholm, SE    360.64 ms   0.0%    0.44 Mbps      201.32 Mbps    i3D.net - Stockholm
 Oslo, NO         371.58 ms   0.0%    1.26 Mbps      205.46 Mbps    Telia Norge AS - Oslo
 Moscow, RU       410.73 ms   0.0%    212.83 Mbps    193.05 Mbps    MTS - Moscow
 Petersburg, RU   400.02 ms   0.0%    243.48 Mbps    206.53 Mbps    MegaFon - Saint Petersburg
 Istanbul, TR     341.64 ms   0.0%    3.60 Mbps      234.38 Mbps    Turkcell - Istanbul
 Tbilisi, GE      368.47 ms   0.0%    2047.49 Mbps   202.74 Mbps    Silknet JSC - Tbilisi
---------------------------------------------------------------------------

[osobo]

@girish Sorry for the delayed response. My server is also hosted in Singapore (by Hostinger). I've noted a couple of seconds delay to open Cloudron's app store but it is definitively back. Thank you again for the support !