Two small manifest fields for slow-starting and long-running apps (AI servers especially)

[LoudLemur]

We package several AI servers for the community store (text embeddings, reranking, speech, LLM inference). Two platform limits come up in almost every one, and both look cheap to lift. They sit alongside the gpu capability (topic 15756) and /dev/shm size (topic 15844) you have already agreed to, and are not a bump of either.

1. A start period for the health check. Several model servers only bind their HTTP port after warming up the model, which takes tens of seconds on a CPU. During that window healthCheckPath gets connection-refused rather than a 503, and the app restart-loops before it is ever ready. A local docker run never shows this, because Docker does not health-check during warm-up, so it only appears on a real box. Our workaround is an nginx shim inside the image that answers /health at once and proxies everything else. It works, but every package has to rediscover it. Suggested shape: an optional healthCheckStartPeriod in seconds, like Docker's own HEALTHCHECK --start-period, during which failures are not counted.

2. A per-app proxy read timeout. The reverse proxy cuts a request at about 60 seconds, and it cannot be set per app. A cold-model inference call, a long summary or a large synchronous job hits it, while the app's own server (gunicorn at 600 seconds, for example) would have waited. Streaming responses help where the app supports them, but many endpoints do not stream. Suggested shape: an optional proxyReadTimeout in seconds, with today's value as the default and a sensible maximum.

Both are optional, both default to today's behaviour, and older boxes can ignore them, like other unknown fields.

[james]

Hello @loudlemur

@LoudLemur said:

the app restart-loops before it is ever ready

A failing health check does not cause the app to restart, there must be something else going on.

[LoudLemur]

@james said:

A failing health check does not cause the app to restart, there must be something else going on.

@james You are right, thank you. We rechecked: the same change that added our shim also raised the memory limit, and the container was exiting 137, an OOM kill. The memory was the fix, not the shim. Please disregard request 1.

Request 2 stands. A slow cold-model call completes via localhost inside the container, but through the reverse proxy it is cut at 60 seconds every time.

[james]

Hello @loudlemur

@LoudLemur said:

Request 2 stands. A slow cold-model call completes via localhost inside the container, but through the reverse proxy it is cut at 60 seconds every time.

Just raising the timeout higher is not a resilient approach.
One system might have a response time of 30s and on another maybe 60s+
So no matter to what value the timeout is set, it can always reach the timeout.

This sounds more like an design issue of the application.
If something is still working it should inform the pending request about that and not simply do nothing until a timeout is reached.

Something similar we just hit with https://forum.cloudron.io/topic/15914/large-app-backup-70gb-fails-at-rotate-copy-step-with-nosuchkey-ionos-s3-read-after-write-consistency-but-cloudron-has-no-retry
Where the IONOS proxy in front of the S3 server simply did not respond, reached the timeout of 600s while it was in reality still working.
This was because IONOS did not implement CompleteMultipartUpload of S3 correctly.
While the CompleteMultipartUpload request could take several minutes to finalize and that is why S3 should periodically sends white space characters to keep the connection from timing out.
They did not do that and thus proxy timeout was reached and caused all sort of issues.
They could have increased the timeout to 1000s, 2000s or what ever value, but it would not have fixed the actual issue.

The same can be applied here.
If a slow cold-model takes time to complete, it should report it is still working instead of doing nothing causing the proxy to throw the timeout.

[LoudLemur]

Thank you, @james. You were right, and more right than we knew: please disregard request 2 as well.

Before replying, we measured it instead of repeating our notes.

• The platform does not cut at 60 seconds. The app proxy in nginxconfig.ejs sets proxy_read_timeout 3500, and it has done so since at least 8.3.0.
• A test confirms it. On 10.0.4 we installed a small app whose endpoint stays completely silent before answering. Through the reverse proxy it answered 200 after 30, 75 and 150 seconds.
• The 60 seconds we saw came from a client's own timeout. We blamed the proxy because a call to localhost worked, and that was an inference we never tested. It then spread through our notes into this request. Sorry for the noise, twice in one thread.

Your design point stands, and the IONOS example makes it well: a longer timeout only moves the wall. Where an upstream can stream or report progress, our packages will say so and default to it, because real clients do time out. We are also correcting the documentation in our packages that repeated the wrong cause.

[james]

Hello @loudlemur

@LoudLemur said:

Sorry for the noise, twice in one thread.

No problem and I don't mind it at all.
It is always good to exchange thoughts which leads to more knowledge for everyone and this also creates better and more stable software.