Errors in log and net responding clients

[dima]

Hi,

since a couple of days we have problems with vaultwarden. Logs show:
Mar 13 10:13:17 => Healtheck error: Error: Timeout of 7000ms exceeded
Mar 13 10:13:27 => Healtheck error: Error: Timeout of 7000ms exceeded
Mar 13 10:13:37 => Healtheck error: Error: Timeout of 7000ms exceeded
Mar 13 10:13:47 => Healtheck error: Error: Timeout of 7000ms exceeded
Mar 13 10:13:57 => Healtheck error: Error: Timeout of 7000ms exceeded

and Clients can not update or login.

I set the memory limit to 5 Gb with no success. Performance graphs are all at low state.

Someone with a glue or advise?

Regards
Dirk

[Ixapack]

Hello
up , i have exactly the same issue
Look like for me the instance freeze after sending 2FA mail
No clue at the moment
regards

[nebulon]

Do you see any app startup errors in the logs when you restart the app?

[dima]

I can not see any abnormal:

Mar 13 12:12:05 => Starting supervisord
Mar 13 12:12:05 box:tasks update 14557: {"percent":100,"message":"Done"}
Mar 13 12:12:05 box:taskworker Task took 7.035 seconds
Mar 13 12:12:05 box:tasks setCompleted - 14557: {"result":null,"error":null}
Mar 13 12:12:05 box:tasks update 14557: {"percent":100,"result":null,"error":null}
Mar 13 12:12:05 2025-03-13 11:12:05,952 CRIT Supervisor is running as root. Privileges were not dropped because no user is specified in the config file. If you intend to run as root, you can set user=root in the config file to avoid this message.
Mar 13 12:12:05 2025-03-13 11:12:05,952 INFO Included extra file "/etc/supervisor/conf.d/apache2.conf" during parsing
Mar 13 12:12:05 2025-03-13 11:12:05,953 INFO Included extra file "/etc/supervisor/conf.d/bitwarden_rs.conf" during parsing
Mar 13 12:12:05 2025-03-13 11:12:05,959 INFO RPC interface 'supervisor' initialized
Mar 13 12:12:05 2025-03-13 11:12:05,960 CRIT Server 'unix_http_server' running without any HTTP authentication checking
Mar 13 12:12:05 2025-03-13 11:12:05,960 INFO supervisord started with pid 1
Mar 13 12:12:06 2025-03-13 11:12:06,965 INFO spawned: 'apache2' with pid 27
Mar 13 12:12:06 2025-03-13 11:12:06,969 INFO spawned: 'core' with pid 28
Mar 13 12:12:06 /--------------------------------------------------------------------
Mar 13 12:12:06 | Starting Vaultwarden |
Mar 13 12:12:06 |--------------------------------------------------------------------|
Mar 13 12:12:06 | This is an unofficial Bitwarden implementation, DO NOT use the |
Mar 13 12:12:06 | official channels to report bugs/features, regardless of client. |
Mar 13 12:12:06 | Send usage/configuration questions or feature requests to: |
Mar 13 12:12:06 | https://github.com/dani-garcia/vaultwarden/discussions or |
Mar 13 12:12:06 | https://vaultwarden.discourse.group/ |
Mar 13 12:12:06 | Report suspected bugs/issues in the software itself at: |
Mar 13 12:12:06 | https://github.com/dani-garcia/vaultwarden/issues/new |
Mar 13 12:12:06 --------------------------------------------------------------------/
Mar 13 12:12:06 2025-03-13T11:12:06Z
Mar 13 12:12:06 [INFO] Using saved config from /app/data/config.json for configuration.
Mar 13 12:12:06 2025-03-13T11:12:06Z
Mar 13 12:12:06 [NOTICE] You are using a plain text ADMIN_TOKEN which is insecure.
Mar 13 12:12:06 Please generate a secure Argon2 PHC string by using vaultwarden hash or argon2.
Mar 13 12:12:06 See: https://github.com/dani-garcia/vaultwarden/wiki/Enabling-admin-page#secure-the-admin_token
Mar 13 12:12:06 2025-03-13T11:12:06Z
Mar 13 12:12:07 [2025-03-13 11:12:07.204][start][INFO] Rocket has launched from http://127.0.0.1:3000
Mar 13 12:12:07 [Thu Mar 13 11:12:07.270937 2025] [mpm_prefork:notice] [pid 43] AH00163: Apache/2.4.52 (Ubuntu) mod_perl/2.0.12 Perl/v5.34.0 configured -- resuming normal operations
Mar 13 12:12:07 [Thu Mar 13 11:12:07.271015 2025] [core:notice] [pid 43] AH00094: Command line: '/usr/sbin/apache2 -D FOREGROUND'
Mar 13 12:12:08 2025-03-13 11:12:08,272 INFO success: apache2 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Mar 13 12:12:08 2025-03-13 11:12:08,273 INFO success: core entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Mar 13 12:12:08 [2025-03-13 11:12:08.613][request][INFO] GET /notifications/hub?access_token=eyJ0eXAiOiJKV1QiL
Mar 13 12:12:08 [2025-03-13 11:12:08.613][vaultwarden::api::notifications][INFO] Accepting Rocket WS connection from 217.5.161.14
Mar 13 12:12:08 [2025-03-13 11:12:08.614][response][INFO] (websockets_hub) GET /notifications/hub?<data..> => 200 OK

[joseph]

The processes seem running fine from the logs. So, the healthcheck errors appear after the above logs you posted?

Can you open a web terminal and check the file /run/vaultwarden/vaultwarden.log ?

[dima]

Mar 13 14:12:15 [2025-03-13 13:12:15.157][response][INFO] (websockets_hub) GET /notifications/hub?<data..> => 200 OK
Mar 13 14:12:17 => Healtheck error: Error: Timeout of 7000ms exceeded
Mar 13 14:12:27 => Healtheck error: Error: Timeout of 7000ms exceeded
Mar 13 14:12:37 => Healtheck error: Error: Timeout of 7000ms exceeded
Mar 13 14:12:47 => Healtheck error: Error: Timeout of 7000ms exceeded
Mar 13 14:12:57 => Healtheck error: Error: Timeout of 7000ms exceeded
Mar 13 14:13:07 => Healtheck error: Error: Timeout of 7000ms exceeded
Mar 13 14:13:14 [2025-03-13 13:13:14.017][vaultwarden::api::notifications][INFO] Closing WS connection from

and vaultwarden.log:
[2025-03-13 13:12:15.157][response][INFO] (websockets_hub) GET /notifications/hub?<data..> => 200 OK
[2025-03-13 13:13:14.017][vaultwarden::api::notifications][INFO] Closing WS connection from

there is a gap ..

[dima]

can you increase the timeout?

[Ixapack]

@nebulon Hello same as dima . i don't have any issue and suddently timeout

[joseph]

Can you install a fresh Vaultwarden and check if that timesout ? I am wondering if you guys enabled some feature in Vaultwarden that is now making the healthchecks fail. It's not a timeout issue afaict (7 secs is plenty enough for a simple healtchech call)

[dima]

you mention a 2nd instance without data?

[joseph]

@dima yes, correct, just a test instance without data to see if that works. Atleast, this will rule out networking/database etc

Do you happen to know/remember if you changed any setting in Vaultwarden config settings? BTW, the http port (if this is somwhere in the ui page) has to be 8000.

[Ixapack]

Hello

here below my config .
I enable yubico and the 2fa and have been powned
also the token on argo2

I try right now a new instance

{
"domain": "https://vault.ixapack.com",
"sends_allowed": true,
"hibp_api_key": "",
"incomplete_2fa_time_limit": 3,
"disable_icon_download": false,
"signups_allowed": true,
"signups_verify": false,
"signups_verify_resend_time": 3600,
"signups_verify_resend_limit": 6,
"signups_domains_whitelist": "ixapack.com",
"org_creation_users": "informatique@ixapack.com",
"invitations_allowed": true,
"emergency_access_allowed": true,
"email_change_allowed": false,
"password_iterations": 600000,
"password_hints_allowed": true,
"show_password_hint": false,
"admin_token": "$argon2id$",
"invitation_org_name": "Ixapass",
"ip_header": "X-Forwarded-For",
"icon_redirect_code": 302,
"icon_cache_ttl": 2592000,
"icon_cache_negttl": 259200,
"icon_download_timeout": 10,
"http_request_block_non_global_ips": true,
"disable_2fa_remember": false,
"authenticator_disable_time_drift": false,
"require_device_email": false,
"reload_templates": false,
"log_timestamp_format": "%Y-%m-%d %H:%M:%S.%3f",
"admin_session_lifetime": 20,
"increase_note_size_limit": false,
"_enable_yubico": true,
"yubico_client_id": "",
"yubico_secret_key": "",
"_enable_duo": false,
"_enable_smtp": true,
"use_sendmail": false,
"smtp_host": "mail",
"smtp_security": "off",
"smtp_port": 2525,
"smtp_from": "vault.app@ixapack.io",
"smtp_from_name": "noreply",
"smtp_username": "vault.app@ixapack.io",
"smtp_password": "",
"smtp_auth_mechanism": "Plain",
"smtp_timeout": 15,
"smtp_embed_images": true,
"smtp_accept_invalid_certs": true,
"smtp_accept_invalid_hostnames": true,
"_enable_email_2fa": true,
"email_token_size": 6,
"email_expiration_time": 600,
"email_attempts_limit": 3,
"email_2fa_enforce_on_verified_invite": true,
"email_2fa_auto_fallback": true
}

[Ixapack]

New instance i don't have any down
I try on my old one to disable one by one some function still the time out .
odd

[dima]

no errors in the test instance - healthcheck every 10 sec

[dima]

Any news?

Also without significant load, it isnt fine

10:00 healthmonitor App is down
23:31 healthmonitor App is back online
23:28 healthmonitor App is down
23:08 healthmonitor App is back online
21:48 healthmonitor App is down
20:06 healthmonitor App is back online
20:05 healthmonitor App is down

[joseph]

@dima @Ixapack please contact us on support@cloudron.io . It's unclear if the issue is with the app or the package or the platform

[girish]

@dima @Ixapack the issue seems to be that apache is getting busy because of many websocket connections. since the number of apache instances gets saturated, the healthcheck call stops working (since there is free apache process to respond).

I checked upstream . Looks like they have added a /alive route and we don't need the internal apache anymore . https://git.cloudron.io/packages/vaultwarden-app/-/merge_requests/14 . Think that should sort out the issue.

[Ixapack]

@girish Hello i confirm, the update fix the issue
thank you