The turn service ran out of memory

[girish]

@nebulon said in The turn service ran out of memory:

Maybe if you have overprovisioned your system, the kernel decided to kill turn?

Want to re-iterate this... The OOM Killer of the kernel will randomly kill processes when it needs memory for a process. So, sometimes it will say turn ran out of memory even if you never used turn because it killed turn.

You can look into the dmesg output, I think you will find some oom info there.

[warg]

In dmesg I don't see relevant-looking memory errors like out of RAM. I checked /var/log/kern.log then and I just see tons of package drops (likely from firewall/fail2ban).

Regarding OOM killer: I doubt it. There was no user activity and according to system info page, there's plenty of free RAM for the last 7 days:

I just noticed turn is again at 99 % RAM.

[warg]

I just checked the error log of turn now (before it was nearly completely empty, likely due to reboot/kill of app) and it shows tons of errors (just a few of 35k lines):

2023-07-26T12:08:13.000Z udp send: Invalid argument
2023-07-26T12:08:13.000Z udp send: Invalid argument
2023-07-26T12:08:13.000Z udp send: Invalid argument
2023-07-26T12:08:13.000Z udp send: Invalid argument
2023-07-26T12:08:13.000Z udp send: Invalid argument
2023-07-26T12:08:13.000Z udp send: Invalid argument
2023-07-26T12:08:13.000Z udp send: Invalid argument
2023-07-26T12:08:13.000Z udp send: Invalid argument
2023-07-26T12:08:13.000Z udp send: Invalid argument
2023-07-26T12:08:13.000Z udp send: Invalid argument
2023-07-26T12:08:13.000Z udp send: Invalid argument
2023-07-26T12:08:13.000Z udp send: Invalid argument
2023-07-26T12:08:13.000Z udp send: Invalid argument
2023-07-26T12:08:13.000Z udp send: Invalid argument
2023-07-26T12:08:13.000Z udp send: Invalid argument
2023-07-26T12:08:13.000Z udp send: Invalid argument
2023-07-26T12:08:13.000Z udp send: Invalid argument
2023-07-26T12:08:13.000Z udp send: Invalid argument
2023-07-26T12:08:13.000Z udp send: Invalid argument
2023-07-26T12:08:14.000Z 151291: : ERROR: handle_udp_packet: socket to be closed
2023-07-26T12:08:14.000Z 151291: : ERROR: handle_udp_packet: socket to be closed
2023-07-26T12:08:14.000Z 151291: : ERROR: handle_udp_packet: socket to be closed
2023-07-26T12:08:14.000Z 151291: : ERROR: handle_udp_packet: socket to be closed
2023-07-26T12:08:14.000Z 151291: : ERROR: handle_udp_packet: socket to be closed
2023-07-26T12:08:14.000Z 151291: : ERROR: handle_udp_packet: socket to be closed
2023-07-26T12:08:14.000Z 151291: : ERROR: handle_udp_packet: socket to be closed
2023-07-26T12:08:14.000Z 151291: : ERROR: handle_udp_packet: socket to be closed
2023-07-26T12:08:14.000Z 151291: : ERROR: handle_udp_packet: socket to be closed
2023-07-26T12:08:14.000Z 151291: : ERROR: handle_udp_packet: socket to be closed
2023-07-26T12:08:14.000Z 151291: : handle_udp_packet: 111.111: thrid=0x7f966bfff640: Amap = 0x7f9668a697b0, socket container=0x7f9668a697b0, local addr <removed>:3478, remote addr <removed>, s=0x55bd58976f00, done=0, tbc=1
2023-07-26T12:08:14.000Z 151291: : handle_udp_packet: 111.111: thrid=0x7f966bfff640: Amap = 0x7f9668a697b0, socket container=0x7f9668a697b0, local addr <removed>:3478, remote addr <removed>, s=0x55bd58976f00, done=0, tbc=1
2023-07-26T12:08:14.000Z 151291: : handle_udp_packet: 111.111: thrid=0x7f966bfff640: Amap = 0x7f9668a697b0, socket container=0x7f9668a697b0, local addr <removed>:3478, remote addr <removed>, s=0x55bd58976f00, done=0, tbc=1
 111.111: thrid=0x7f966bfff640: Amap = 0x7f9668a697b0, socket container=0x7f9668a697b0, local addr <removed>:3478, remote addr <removed>, s=0x55bd58976f00, done=0, tbc=1
2023-07-26T12:08:14.000Z 151291: : handle_udp_packet: 111.111: thrid=0x7f966bfff640: Amap = 0x7f9668a697b0, socket container=0x7f9668a697b0, local addr <removed>:3478, remote addr <removed>, s=0x55bd58976f00, done=0, tbc=1
2023-07-26T12:08:14.000Z 151291: : handle_udp_packet: 111.111: thrid=0x7f966bfff640: Amap = 0x7f9668a697b0, socket container=0x7f9668a697b0, local addr <removed>:3478, remote addr <removed>, s=0x55bd58976f00, done=0, tbc=1
2023-07-26T12:08:14.000Z 151291: : handle_udp_packet: 111.111: thrid=0x7f966bfff640: Amap = 0x7f9668a697b0, socket container=0x7f9668a697b0, local addr <removed>:3478, remote addr <removed>, s=0x55bd58976f00, done=0, tbc=1
2023-07-26T12:08:14.000Z 151291: : handle_udp_packet: 111.111: thrid=0x7f966bfff640: Amap = 0x7f9668a697b0, socket container=0x7f9668a697b0, local addr <removed>:3478, remote addr <removed>, s=0x55bd58976f00, done=0, tbc=1
2023-07-26T12:08:14.000Z 151291: : handle_udp_packet: 111.111: thrid=0x7f966bfff640: Amap = 0x7f9668a697b0, socket container=0x7f9668a697b0, local addr <removed>:3478, remote addr <removed>, s=0x55bd58976f00, done=0, tbc=1
2023-07-26T12:08:14.000Z 151291: : handle_udp_packet: 111.111: thrid=0x7f966bfff640: Amap = 0x7f9668a697b0, socket container=0x7f9668a697b0, local addr <removed>:3478, remote addr <removed>, s=0x55bd58976f00, done=0, tbc=1
2023-07-26T12:08:14.000Z 151292: : ERROR: handle_udp_packet: socket to be closed
2023-07-26T12:08:14.000Z 151292: : ERROR: handle_udp_packet: socket to be closed
2023-07-26T12:08:14.000Z 151292: : handle_udp_packet: 111.111: thrid=0x7f966bfff640: Amap = 0x7f9668a697b0, socket container=0x7f9668a697b0, local addr <removed>:3478, remote addr <removed>, s=0x55bd58976f00, done=0, tbc=1
2023-07-26T12:08:14.000Z 151292: : handle_udp_packet: 111.111: thrid=0x7f966bfff640: Amap = 0x7f9668a697b0, socket container=0x7f9668a697b0, local addr <removed>:3478, remote addr <removed>, s=0x55bd58976f00, done=0, tbc=1
2023-07-26T12:08:14.000Z udp send: Invalid argument
2023-07-26T12:08:14.000Z udp send: Invalid argument
2023-07-26T12:08:14.000Z udp send: Invalid argument
2023-07-26T12:08:14.000Z udp send: Invalid argument
2023-07-26T12:08:14.000Z udp send: Invalid argument
2023-07-26T12:08:14.000Z udp send: Invalid argument

[girish]

@warg are you using the apps that use turn ?

Tried searching for these error messages from coturn but I am not seeing getting any leads so far.

[warg]

I never heard of turn protocol/daemon before so I don't know if and which app uses it internally. Actively (= I'm aware of it): Nope. These apps are installed on my Cloudron v7.5.0:

• bookstack
• collabora
• lamp
• n8n
• nextcloud
• nocodb

These were recently removed:

• LibreTranslate
• Cloudron Build Service

[warg]

I just rebooted the server to apply kernel updates from livepatching and I see errors like this from the startup (I enabled IPv6 on Cloudron short-term but I'm not sure it's relevant for this issue):

2023-07-27T16:16:09.000Z 0: : IO method (general relay thread): epoll (with changelist)
2023-07-27T16:16:09.000Z 0: : IO method (general relay thread): epoll (with changelist)
2023-07-27T16:16:09.000Z 0: : IO method (general relay thread): epoll (with changelist)
2023-07-27T16:16:09.000Z 0: : IO method (general relay thread): epoll (with changelist)
2023-07-27T16:16:09.000Z 0: : IO method (general relay thread): epoll (with changelist)
2023-07-27T16:16:09.000Z 0: : IO method (general relay thread): epoll (with changelist)
2023-07-27T16:16:09.000Z 0: : IO method (general relay thread): epoll (with changelist)
2023-07-27T16:16:09.000Z 0: : IO method (general relay thread): epoll (with changelist)
2023-07-27T16:16:09.000Z 0: : IO method (general relay thread): epoll (with changelist)
2023-07-27T16:16:09.000Z 0: : IO method (general relay thread): epoll (with changelist)
2023-07-27T16:16:09.000Z 0: : IO method (general relay thread): epoll (with changelist)
2023-07-27T16:16:09.000Z 0: : IO method (general relay thread): epoll (with changelist)
2023-07-27T16:16:09.000Z 0: : IO method (main listener thread): epoll (with changelist)
2023-07-27T16:16:09.000Z 0: : IO method (main listener thread): epoll (with changelist)
2023-07-27T16:16:09.000Z 0: : Listener address to use: 127.0.0.1
2023-07-27T16:16:09.000Z 0: : Listener address to use: 127.0.0.1
2023-07-27T16:16:09.000Z 0: : Listener address to use: 172.17.0.1
2023-07-27T16:16:09.000Z 0: : Listener address to use: 172.17.0.1
2023-07-27T16:16:09.000Z 0: : Listener address to use: 172.18.0.1
2023-07-27T16:16:09.000Z 0: : Listener address to use: 172.18.0.1
2023-07-27T16:16:09.000Z 0: : Listener address to use: <removed>
2023-07-27T16:16:09.000Z 0: : Listener address to use: <removed>
2023-07-27T16:16:09.000Z 0: : Listener address to use: <removed>
2023-07-27T16:16:09.000Z 0: : Listener address to use: <removed>
2023-07-27T16:16:09.000Z 0: : Listener address to use: ::1
2023-07-27T16:16:09.000Z 0: : Listener address to use: ::1
2023-07-27T16:16:09.000Z 0: : Listener address to use: <removed>
2023-07-27T16:16:09.000Z 0: : Listener address to use: <removed>
2023-07-27T16:16:09.000Z 0: : MongoDB is not supported
2023-07-27T16:16:09.000Z 0: : MongoDB is not supported
2023-07-27T16:16:09.000Z 0: : MySQL supported
2023-07-27T16:16:09.000Z 0: : MySQL supported
2023-07-27T16:16:09.000Z 0: : NO EXPLICIT LISTENER ADDRESS(ES) ARE CONFIGURED
2023-07-27T16:16:09.000Z 0: : NO EXPLICIT LISTENER ADDRESS(ES) ARE CONFIGURED
2023-07-27T16:16:09.000Z 0: : NO EXPLICIT RELAY ADDRESS(ES) ARE CONFIGURED
2023-07-27T16:16:09.000Z 0: : NO EXPLICIT RELAY ADDRESS(ES) ARE CONFIGURED
2023-07-27T16:16:09.000Z 0: : OpenSSL compile-time version: OpenSSL 3.0.2 15 Mar 2022 (0x30000020)
2023-07-27T16:16:09.000Z 0: : OpenSSL compile-time version: OpenSSL 3.0.2 15 Mar 2022 (0x30000020)
2023-07-27T16:16:09.000Z 0: : PostgreSQL supported
2023-07-27T16:16:09.000Z 0: : PostgreSQL supported
2023-07-27T16:16:09.000Z 0: : Redis supported
2023-07-27T16:16:09.000Z 0: : Redis supported
2023-07-27T16:16:09.000Z 0: : Relay address to use: 172.17.0.1
2023-07-27T16:16:09.000Z 0: : Relay address to use: 172.17.0.1
2023-07-27T16:16:09.000Z 0: : Relay address to use: 172.18.0.1
2023-07-27T16:16:09.000Z 0: : Relay address to use: 172.18.0.1
2023-07-27T16:16:09.000Z 0: : Relay address to use: <removed>
2023-07-27T16:16:09.000Z 0: : Relay address to use: <removed>
2023-07-27T16:16:09.000Z 0: : Relay address to use: <removed>
2023-07-27T16:16:09.000Z 0: : Relay address to use: <removed>
2023-07-27T16:16:09.000Z 0: : Relay address to use: <removed>
2023-07-27T16:16:09.000Z 0: : Relay address to use: <removed>
2023-07-27T16:16:09.000Z 0: : Relay ports initialization done
2023-07-27T16:16:09.000Z 0: : Relay ports initialization done
2023-07-27T16:16:09.000Z 0: : SQLite supported, default database location is /var/lib/turn/turndb
2023-07-27T16:16:09.000Z 0: : SQLite supported, default database location is /var/lib/turn/turndb
2023-07-27T16:16:09.000Z 0: : SSL23: Certificate file found: /run/turnserver/turn.cloudron.cert
2023-07-27T16:16:09.000Z 0: : SSL23: Certificate file found: /run/turnserver/turn.cloudron.cert
2023-07-27T16:16:09.000Z 0: : SSL23: Private key file found: /run/turnserver/turn.cloudron.key
2023-07-27T16:16:09.000Z 0: : SSL23: Private key file found: /run/turnserver/turn.cloudron.key
2023-07-27T16:16:09.000Z 0: : TLS cipher suite: ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384
2023-07-27T16:16:09.000Z 0: : TLS cipher suite: ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384
2023-07-27T16:16:09.000Z 0: : TLS supported
2023-07-27T16:16:09.000Z 0: : TLS supported
2023-07-27T16:16:09.000Z 0: : TLS1.2: Certificate file found: /run/turnserver/turn.cloudron.cert
2023-07-27T16:16:09.000Z 0: : TLS1.2: Certificate file found: /run/turnserver/turn.cloudron.cert
2023-07-27T16:16:09.000Z 0: : TLS1.2: Private key file found: /run/turnserver/turn.cloudron.key
2023-07-27T16:16:09.000Z 0: : TLS1.2: Private key file found: /run/turnserver/turn.cloudron.key
2023-07-27T16:16:09.000Z 0: : TURN/STUN ALPN supported
2023-07-27T16:16:09.000Z 0: : TURN/STUN ALPN supported
2023-07-27T16:16:09.000Z 0: : Third-party authorization (oAuth) supported
2023-07-27T16:16:09.000Z 0: : Third-party authorization (oAuth) supported
2023-07-27T16:16:09.000Z 0: : Total: 5 'real' addresses discovered
2023-07-27T16:16:09.000Z 0: : Total: 5 'real' addresses discovered
2023-07-27T16:16:09.000Z 0: : Total: 5 relay addresses discovered
2023-07-27T16:16:09.000Z 0: : Total: 5 relay addresses discovered
2023-07-27T16:16:09.000Z 0: : Trying to bind DTLS/UDP listener socket to addr [<removed>]:3478, again...
2023-07-27T16:16:09.000Z 0: : Trying to bind DTLS/UDP listener socket to addr [<removed>]:3478, again...
2023-07-27T16:16:09.000Z 0: : Trying to bind TLS/TCP listener socket to addr [<removed>]:3478, again...
2023-07-27T16:16:09.000Z 0: : Trying to bind TLS/TCP listener socket to addr [<removed>]:3478, again...
2023-07-27T16:16:09.000Z 0: : Trying to bind TLS/TCP listener socket to addr [<removed>]:3478, again...
2023-07-27T16:16:09.000Z 0: : Trying to bind TLS/TCP listener socket to addr [<removed>]:3478, again...
2023-07-27T16:16:09.000Z 0: : Trying to bind TLS/TCP listener socket to addr [<removed>]:3478, again...
2023-07-27T16:16:09.000Z 0: : Trying to bind TLS/TCP listener socket to addr [<removed>]:3478, again...
2023-07-27T16:16:09.000Z 0: : Trying to bind TLS/TCP listener socket to addr [<removed>]:3478, again...
2023-07-27T16:16:09.000Z 0: : Trying to bind TLS/TCP listener socket to addr [<removed>]:3478, again...
2023-07-27T16:16:09.000Z 0: : Trying to bind TLS/TCP listener socket to addr [<removed>]:3478, again...
2023-07-27T16:16:09.000Z 0: : Trying to bind TLS/TCP listener socket to addr [<removed>]:3478, again...
2023-07-27T16:16:09.000Z 0: : Trying to bind TLS/TCP listener socket to addr [<removed>]:3478, again...
2023-07-27T16:16:09.000Z 0: : Trying to bind TLS/TCP listener socket to addr [<removed>]:3478, again...
2023-07-27T16:16:09.000Z 0: : Trying to bind fd 175 to <[<removed>]:3478>: errno=99
2023-07-27T16:16:09.000Z 0: : Trying to bind fd 175 to <[<removed>]:3478>: errno=99
2023-07-27T16:16:09.000Z 0: : Trying to bind fd 249 to <[<removed>]:3478>: errno=99
2023-07-27T16:16:09.000Z 0: : Trying to bind fd 249 to <[<removed>]:3478>: errno=99
2023-07-27T16:16:09.000Z 0: : Trying to bind fd 309 to <[<removed>]:3478>: errno=99
2023-07-27T16:16:09.000Z 0: : Trying to bind fd 309 to <[<removed>]:3478>: errno=99
2023-07-27T16:16:09.000Z 0: : Trying to bind fd 363 to <[<removed>]:3478>: errno=99
2023-07-27T16:16:09.000Z 0: : Trying to bind fd 363 to <[<removed>]:3478>: errno=99
2023-07-27T16:16:09.000Z 0: : Trying to bind fd 381 to <[<removed>]:3478>: errno=99
2023-07-27T16:16:09.000Z 0: : Trying to bind fd 381 to <[<removed>]:3478>: errno=99
2023-07-27T16:16:09.000Z 0: : Trying to bind fd 458 to <[<removed>]:3478>: errno=99
2023-07-27T16:16:09.000Z 0: : Trying to bind fd 458 to <[<removed>]:3478>: errno=99
2023-07-27T16:16:09.000Z 0: : Trying to bind fd 469 to <[<removed>]:3478>: errno=99
2023-07-27T16:16:09.000Z 0: : Trying to bind fd 469 to <[<removed>]:3478>: errno=99
2023-07-27T16:16:09.000Z 0: : Wait for relay ports initialization...
2023-07-27T16:16:09.000Z 0: : Wait for relay ports initialization...
2023-07-27T16:16:09.000Z 0: : pid file created: /run/turnserver/turnserver.pid
2023-07-27T16:16:09.000Z 0: : pid file created: /run/turnserver/turnserver.pid
2023-07-27T16:16:09.000Z 0: : turn server id=0 created
2023-07-27T16:16:09.000Z 0: : turn server id=0 created
2023-07-27T16:16:09.000Z 0: : turn server id=1 created
2023-07-27T16:16:09.000Z 0: : turn server id=1 created
2023-07-27T16:16:09.000Z 0: : turn server id=2 created
2023-07-27T16:16:09.000Z 0: : turn server id=2 created
2023-07-27T16:16:09.000Z 0: : turn server id=3 created
2023-07-27T16:16:09.000Z 0: : turn server id=3 created
2023-07-27T16:16:09.000Z 0: : turn server id=4 created
2023-07-27T16:16:09.000Z 0: : turn server id=4 created
2023-07-27T16:16:09.000Z 0: : turn server id=5 created
2023-07-27T16:16:09.000Z 0: : turn server id=5 created
2023-07-27T16:16:09.000Z ==== Show him the instruments, Practical Frost: ====
2023-07-27T16:16:09.000Z ==== Show him the instruments, Practical Frost: ====
2023-07-27T16:16:09.000Z =====================================================
2023-07-27T16:16:09.000Z =====================================================
2023-07-27T16:16:09.000Z => Ensure permissions
2023-07-27T16:16:09.000Z => Generating turnserver.conf
2023-07-27T16:16:09.000Z => Starting coturn
2023-07-27T16:16:09.000Z Be aware that you could not mix the username/password and the shared secret based auth methods.
2023-07-27T16:16:09.000Z Be aware that you could not mix the username/password and the shared secret based auth methods.
2023-07-27T16:16:09.000Z CONFIGURATION ALERT: You specified --lt-cred-mech and --use-auth-secret in the same time.
2023-07-27T16:16:09.000Z CONFIGURATION ALERT: You specified --lt-cred-mech and --use-auth-secret in the same time.
2023-07-27T16:16:09.000Z Cannot bind local socket to addr: Cannot assign requested address
2023-07-27T16:16:09.000Z Cannot bind local socket to addr: Cannot assign requested address
2023-07-27T16:16:09.000Z Cannot bind local socket to addr: Cannot assign requested address
2023-07-27T16:16:09.000Z Cannot bind local socket to addr: Cannot assign requested address
2023-07-27T16:16:09.000Z Cannot bind local socket to addr: Cannot assign requested address
2023-07-27T16:16:09.000Z Cannot bind local socket to addr: Cannot assign requested address
2023-07-27T16:16:09.000Z Cannot bind local socket to addr: Invalid argument
2023-07-27T16:16:09.000Z Due to the open files/sockets limitation,
2023-07-27T16:16:09.000Z Due to the open files/sockets limitation,
2023-07-27T16:16:09.000Z Max number of open files/sockets allowed for this process: 1048576
2023-07-27T16:16:09.000Z Max number of open files/sockets allowed for this process: 1048576
2023-07-27T16:16:09.000Z RFC 3489/5389/5766/5780/6062/6156 STUN/TURN Server
2023-07-27T16:16:09.000Z RFC 3489/5389/5766/5780/6062/6156 STUN/TURN Server
2023-07-27T16:16:09.000Z Shared secret overrides username/password based auth method. Check your configuration!
2023-07-27T16:16:09.000Z Shared secret overrides username/password based auth method. Check your configuration!
2023-07-27T16:16:09.000Z Version Coturn-4.5.2 'dan Eider'
2023-07-27T16:16:09.000Z Version Coturn-4.5.2 'dan Eider'
2023-07-27T16:16:09.000Z bind: Cannot assign requested address
2023-07-27T16:16:09.000Z bind: Cannot assign requested address
2023-07-27T16:16:09.000Z bind: Cannot assign requested address
2023-07-27T16:16:09.000Z bind: Cannot assign requested address
2023-07-27T16:16:09.000Z bind: Cannot assign requested address
2023-07-27T16:16:09.000Z bind: Cannot assign requested address
2023-07-27T16:16:09.000Z bind: Cannot assign requested address
2023-07-27T16:16:09.000Z max supported number of TURN Sessions possible is: 524000 (approximately)
2023-07-27T16:16:09.000Z max supported number of TURN Sessions possible is: 524000 (approximately)
2023-07-27T16:16:10.000Z 0: : IO method (auth thread): epoll (with changelist)
2023-07-27T16:16:10.000Z 0: : IO method (auth thread): epoll (with changelist)
2023-07-27T16:16:10.000Z 0: : Total General servers: 6
2023-07-27T16:16:10.000Z 0: : Total General servers: 6
2023-07-27T16:16:10.000Z 1: : IO method (admin thread): epoll (with changelist)
2023-07-27T16:16:10.000Z 1: : IO method (admin thread): epoll (with changelist)
2023-07-27T16:16:10.000Z 1: : IO method (auth thread): epoll (with changelist)
2023-07-27T16:16:10.000Z 1: : IO method (auth thread): epoll (with changelist)
2023-07-27T16:16:10.000Z 1: : IO method (auth thread): epoll (with changelist)
2023-07-27T16:16:10.000Z 1: : IO method (auth thread): epoll (with changelist)
2023-07-27T16:16:10.000Z 1: : New GID: turnserver(108)
2023-07-27T16:16:10.000Z 1: : New GID: turnserver(108)
2023-07-27T16:16:10.000Z 1: : New UID: turnserver(106)
2023-07-27T16:16:10.000Z 1: : New UID: turnserver(106)
2023-07-27T16:16:10.000Z 1: : SQLite DB connection success: /var/lib/turn/turndb
2023-07-27T16:16:10.000Z 1: : SQLite DB connection success: /var/lib/turn/turndb

Looks there's some IPv6 IP address binding issue and a max file or socket limit reached error.

[marcusquinn]

Interesting. I had this recently, too. Memory allowance seems to be about 200% more than used. Seems to have righted itself, so just noting in case relevant and others noticed the same recently, too. Only happened once for as long as I can remember, so perhaps nothing, but noting just because occasional things can be hard to understand without evidence and reproducibility.

[warg]

The critical errors with invalid arguments doesn't come up anymore. I think this came up from some Ubuntu Livepatching being done + missing reboot afterwards. Didn't occur the last days and for extra safety I just updated all packages on the server now. Nothing broke (although it's not suggested to do that) and I will see if issue occurs again.

[nebulon]

hopefully critical components like docker were not updated then, since this will eventually break. Make sure to have full backups.

[warg]

Thanks for the hint. I accepted that it might break. So far there were just OS-related components updated and DB-stuff. At the moment I saw no critical errors in the apps and they do work. Cloudron works fine as well at the moment. I also restarted the server to see if somethings breaks on full reload. So far, everything is fine and docker-related stuff wasn't updated.

[warg]

Nope, issue was not related to livepatching it seems. Issue just occured out of nowhere again:

Could this be related to malicious clients, e. g. erros created by bots that try turn-specific exploits?

[girish]

@warg said in The turn service ran out of memory:

Could this be related to malicious clients, e. g. erros created by bots that try turn-specific exploits?

It does seem to be like that.

Is anyone else seeing such messages in the turn service logs?