Errors in log and net responding clients
-
Hi,
since a couple of days we have problems with vaultwarden. Logs show:
Mar 13 10:13:17 => Healtheck error: Error: Timeout of 7000ms exceeded
Mar 13 10:13:27 => Healtheck error: Error: Timeout of 7000ms exceeded
Mar 13 10:13:37 => Healtheck error: Error: Timeout of 7000ms exceeded
Mar 13 10:13:47 => Healtheck error: Error: Timeout of 7000ms exceeded
Mar 13 10:13:57 => Healtheck error: Error: Timeout of 7000ms exceededand Clients can not update or login.
I set the memory limit to 5 Gb with no success. Performance graphs are all at low state.
Someone with a glue or advise?
Regards
Dirk -
I can not see any abnormal:
Mar 13 12:12:05 => Starting supervisord
Mar 13 12:12:05 box:tasks update 14557: {"percent":100,"message":"Done"}
Mar 13 12:12:05 box:taskworker Task took 7.035 seconds
Mar 13 12:12:05 box:tasks setCompleted - 14557: {"result":null,"error":null}
Mar 13 12:12:05 box:tasks update 14557: {"percent":100,"result":null,"error":null}
Mar 13 12:12:05 2025-03-13 11:12:05,952 CRIT Supervisor is running as root. Privileges were not dropped because no user is specified in the config file. If you intend to run as root, you can set user=root in the config file to avoid this message.
Mar 13 12:12:05 2025-03-13 11:12:05,952 INFO Included extra file "/etc/supervisor/conf.d/apache2.conf" during parsing
Mar 13 12:12:05 2025-03-13 11:12:05,953 INFO Included extra file "/etc/supervisor/conf.d/bitwarden_rs.conf" during parsing
Mar 13 12:12:05 2025-03-13 11:12:05,959 INFO RPC interface 'supervisor' initialized
Mar 13 12:12:05 2025-03-13 11:12:05,960 CRIT Server 'unix_http_server' running without any HTTP authentication checking
Mar 13 12:12:05 2025-03-13 11:12:05,960 INFO supervisord started with pid 1
Mar 13 12:12:06 2025-03-13 11:12:06,965 INFO spawned: 'apache2' with pid 27
Mar 13 12:12:06 2025-03-13 11:12:06,969 INFO spawned: 'core' with pid 28
Mar 13 12:12:06 /--------------------------------------------------------------------
Mar 13 12:12:06 | Starting Vaultwarden |
Mar 13 12:12:06 |--------------------------------------------------------------------|
Mar 13 12:12:06 | This is an unofficial Bitwarden implementation, DO NOT use the |
Mar 13 12:12:06 | official channels to report bugs/features, regardless of client. |
Mar 13 12:12:06 | Send usage/configuration questions or feature requests to: |
Mar 13 12:12:06 | https://github.com/dani-garcia/vaultwarden/discussions or |
Mar 13 12:12:06 | https://vaultwarden.discourse.group/ |
Mar 13 12:12:06 | Report suspected bugs/issues in the software itself at: |
Mar 13 12:12:06 | https://github.com/dani-garcia/vaultwarden/issues/new |
Mar 13 12:12:06 --------------------------------------------------------------------/
Mar 13 12:12:06 2025-03-13T11:12:06Z
Mar 13 12:12:06 [INFO] Using saved config from/app/data/config.json
for configuration.
Mar 13 12:12:06 2025-03-13T11:12:06Z
Mar 13 12:12:06 [NOTICE] You are using a plain textADMIN_TOKEN
which is insecure.
Mar 13 12:12:06 Please generate a secure Argon2 PHC string by usingvaultwarden hash
orargon2
.
Mar 13 12:12:06 See: https://github.com/dani-garcia/vaultwarden/wiki/Enabling-admin-page#secure-the-admin_token
Mar 13 12:12:06 2025-03-13T11:12:06Z
Mar 13 12:12:07 [2025-03-13 11:12:07.204][start][INFO] Rocket has launched from http://127.0.0.1:3000
Mar 13 12:12:07 [Thu Mar 13 11:12:07.270937 2025] [mpm_prefork:notice] [pid 43] AH00163: Apache/2.4.52 (Ubuntu) mod_perl/2.0.12 Perl/v5.34.0 configured -- resuming normal operations
Mar 13 12:12:07 [Thu Mar 13 11:12:07.271015 2025] [core:notice] [pid 43] AH00094: Command line: '/usr/sbin/apache2 -D FOREGROUND'
Mar 13 12:12:08 2025-03-13 11:12:08,272 INFO success: apache2 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Mar 13 12:12:08 2025-03-13 11:12:08,273 INFO success: core entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Mar 13 12:12:08 [2025-03-13 11:12:08.613][request][INFO] GET /notifications/hub?access_token=eyJ0eXAiOiJKV1QiL
Mar 13 12:12:08 [2025-03-13 11:12:08.613][vaultwarden::api::notifications][INFO] Accepting Rocket WS connection from 217.5.161.14
Mar 13 12:12:08 [2025-03-13 11:12:08.614][response][INFO] (websockets_hub) GET /notifications/hub?<data..> => 200 OK -
Mar 13 14:12:15 [2025-03-13 13:12:15.157][response][INFO] (websockets_hub) GET /notifications/hub?<data..> => 200 OK
Mar 13 14:12:17 => Healtheck error: Error: Timeout of 7000ms exceeded
Mar 13 14:12:27 => Healtheck error: Error: Timeout of 7000ms exceeded
Mar 13 14:12:37 => Healtheck error: Error: Timeout of 7000ms exceeded
Mar 13 14:12:47 => Healtheck error: Error: Timeout of 7000ms exceeded
Mar 13 14:12:57 => Healtheck error: Error: Timeout of 7000ms exceeded
Mar 13 14:13:07 => Healtheck error: Error: Timeout of 7000ms exceeded
Mar 13 14:13:14 [2025-03-13 13:13:14.017][vaultwarden::api::notifications][INFO] Closing WS connection fromand vaultwarden.log:
[2025-03-13 13:12:15.157][response][INFO] (websockets_hub) GET /notifications/hub?<data..> => 200 OK
[2025-03-13 13:13:14.017][vaultwarden::api::notifications][INFO] Closing WS connection fromthere is a gap ..
-
Can you install a fresh Vaultwarden and check if that timesout ? I am wondering if you guys enabled some feature in Vaultwarden that is now making the healthchecks fail. It's not a timeout issue afaict (7 secs is plenty enough for a simple healtchech call)
-
@dima yes, correct, just a test instance without data to see if that works. Atleast, this will rule out networking/database etc
Do you happen to know/remember if you changed any setting in Vaultwarden config settings? BTW, the http port (if this is somwhere in the ui page) has to be 8000.
-
Hello
here below my config .
I enable yubico and the 2fa and have been powned
also the token on argo2I try right now a new instance
{
"domain": "https://vault.ixapack.com",
"sends_allowed": true,
"hibp_api_key": "",
"incomplete_2fa_time_limit": 3,
"disable_icon_download": false,
"signups_allowed": true,
"signups_verify": false,
"signups_verify_resend_time": 3600,
"signups_verify_resend_limit": 6,
"signups_domains_whitelist": "ixapack.com",
"org_creation_users": "informatique@ixapack.com",
"invitations_allowed": true,
"emergency_access_allowed": true,
"email_change_allowed": false,
"password_iterations": 600000,
"password_hints_allowed": true,
"show_password_hint": false,
"admin_token": "$argon2id$",
"invitation_org_name": "Ixapass",
"ip_header": "X-Forwarded-For",
"icon_redirect_code": 302,
"icon_cache_ttl": 2592000,
"icon_cache_negttl": 259200,
"icon_download_timeout": 10,
"http_request_block_non_global_ips": true,
"disable_2fa_remember": false,
"authenticator_disable_time_drift": false,
"require_device_email": false,
"reload_templates": false,
"log_timestamp_format": "%Y-%m-%d %H:%M:%S.%3f",
"admin_session_lifetime": 20,
"increase_note_size_limit": false,
"_enable_yubico": true,
"yubico_client_id": "",
"yubico_secret_key": "",
"_enable_duo": false,
"_enable_smtp": true,
"use_sendmail": false,
"smtp_host": "mail",
"smtp_security": "off",
"smtp_port": 2525,
"smtp_from": "vault.app@ixapack.io",
"smtp_from_name": "noreply",
"smtp_username": "vault.app@ixapack.io",
"smtp_password": "",
"smtp_auth_mechanism": "Plain",
"smtp_timeout": 15,
"smtp_embed_images": true,
"smtp_accept_invalid_certs": true,
"smtp_accept_invalid_hostnames": true,
"_enable_email_2fa": true,
"email_token_size": 6,
"email_expiration_time": 600,
"email_attempts_limit": 3,
"email_2fa_enforce_on_verified_invite": true,
"email_2fa_auto_fallback": true
} -
Any news?
Also without significant load, it isnt fine
10:00 healthmonitor App is down
23:31 healthmonitor App is back online
23:28 healthmonitor App is down
23:08 healthmonitor App is back online
21:48 healthmonitor App is down
20:06 healthmonitor App is back online
20:05 healthmonitor App is down -
@dima @Ixapack the issue seems to be that apache is getting busy because of many websocket connections. since the number of apache instances gets saturated, the healthcheck call stops working (since there is free apache process to respond).
I checked upstream . Looks like they have added a /alive route and we don't need the internal apache anymore . https://git.cloudron.io/packages/vaultwarden-app/-/merge_requests/14 . Think that should sort out the issue.
-
N nebulon marked this topic as a question
-
N nebulon has marked this topic as solved