Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.


Skip to content
  • Categories
  • Recent
  • Tags
  • Popular
  • Bookmarks
  • Search
Skins
  • Light
  • Cerulean
  • Cosmo
  • Flatly
  • Journal
  • Litera
  • Lumen
  • Lux
  • Materia
  • Minty
  • Morph
  • Pulse
  • Sandstone
  • Simplex
  • Sketchy
  • Spacelab
  • United
  • Yeti
  • Zephyr
  • Dark
  • Cyborg
  • Darkly
  • Quartz
  • Slate
  • Solar
  • Superhero
  • Vapor
  • Default (No Skin)
  • No Skin
Collapse
Brand Logo

Cloudron Forum
Apps | Demo | Docs | Install
  1. Cloudron Forum
  2. Support
  3. The turn service ran out of memory

The turn service ran out of memory

Scheduled Pinned Locked Moved Support
memory
15 Posts 5 Posters 1.0k Views 5 Watching
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • C Offline
      C Offline
      ccfu
      wrote on last edited by
      #2

      Have you checked the logs and also the system information screen?

      1 Reply Last reply
      0
      • nebulonN Offline
        nebulonN Offline
        nebulon
        Staff
        wrote on last edited by
        #3

        One thing to consider here, is that the memory values set in Cloudron are just the upper limits. This is rather unrelated to your actual available memory on the server. So setting high limits will eventually lead to the system killing processes to free up actual physical memory. Maybe if you have overprovisioned your system, the kernel decided to kill turn?

        girishG 1 Reply Last reply
        1
        • nebulonN nebulon

          One thing to consider here, is that the memory values set in Cloudron are just the upper limits. This is rather unrelated to your actual available memory on the server. So setting high limits will eventually lead to the system killing processes to free up actual physical memory. Maybe if you have overprovisioned your system, the kernel decided to kill turn?

          girishG Offline
          girishG Offline
          girish
          Staff
          wrote on last edited by
          #4

          @nebulon said in The turn service ran out of memory:

          Maybe if you have overprovisioned your system, the kernel decided to kill turn?

          Want to re-iterate this... The OOM Killer of the kernel will randomly kill processes when it needs memory for a process. So, sometimes it will say turn ran out of memory even if you never used turn because it killed turn.

          You can look into the dmesg output, I think you will find some oom info there.

          1 Reply Last reply
          0
          • W Offline
            W Offline
            warg
            wrote on last edited by warg
            #5

            In dmesg I don't see relevant-looking memory errors like out of RAM. I checked /var/log/kern.log then and I just see tons of package drops (likely from firewall/fail2ban).

            Regarding OOM killer: I doubt it. There was no user activity and according to system info page, there's plenty of free RAM for the last 7 days:

            grafik.png

            I just noticed turn is again at 99 % RAM.

            1 Reply Last reply
            0
            • W Offline
              W Offline
              warg
              wrote on last edited by
              #6

              I just checked the error log of turn now (before it was nearly completely empty, likely due to reboot/kill of app) and it shows tons of errors (just a few of 35k lines):

              2023-07-26T12:08:13.000Z udp send: Invalid argument
2023-07-26T12:08:13.000Z udp send: Invalid argument
2023-07-26T12:08:13.000Z udp send: Invalid argument
2023-07-26T12:08:13.000Z udp send: Invalid argument
2023-07-26T12:08:13.000Z udp send: Invalid argument
2023-07-26T12:08:13.000Z udp send: Invalid argument
2023-07-26T12:08:13.000Z udp send: Invalid argument
2023-07-26T12:08:13.000Z udp send: Invalid argument
2023-07-26T12:08:13.000Z udp send: Invalid argument
2023-07-26T12:08:13.000Z udp send: Invalid argument
2023-07-26T12:08:13.000Z udp send: Invalid argument
2023-07-26T12:08:13.000Z udp send: Invalid argument
2023-07-26T12:08:13.000Z udp send: Invalid argument
2023-07-26T12:08:13.000Z udp send: Invalid argument
2023-07-26T12:08:13.000Z udp send: Invalid argument
2023-07-26T12:08:13.000Z udp send: Invalid argument
2023-07-26T12:08:13.000Z udp send: Invalid argument
2023-07-26T12:08:13.000Z udp send: Invalid argument
2023-07-26T12:08:13.000Z udp send: Invalid argument
2023-07-26T12:08:14.000Z 151291: : ERROR: handle_udp_packet: socket to be closed
2023-07-26T12:08:14.000Z 151291: : ERROR: handle_udp_packet: socket to be closed
2023-07-26T12:08:14.000Z 151291: : ERROR: handle_udp_packet: socket to be closed
2023-07-26T12:08:14.000Z 151291: : ERROR: handle_udp_packet: socket to be closed
2023-07-26T12:08:14.000Z 151291: : ERROR: handle_udp_packet: socket to be closed
2023-07-26T12:08:14.000Z 151291: : ERROR: handle_udp_packet: socket to be closed
2023-07-26T12:08:14.000Z 151291: : ERROR: handle_udp_packet: socket to be closed
2023-07-26T12:08:14.000Z 151291: : ERROR: handle_udp_packet: socket to be closed
2023-07-26T12:08:14.000Z 151291: : ERROR: handle_udp_packet: socket to be closed
2023-07-26T12:08:14.000Z 151291: : ERROR: handle_udp_packet: socket to be closed
2023-07-26T12:08:14.000Z 151291: : handle_udp_packet: 111.111: thrid=0x7f966bfff640: Amap = 0x7f9668a697b0, socket container=0x7f9668a697b0, local addr <removed>:3478, remote addr <removed>, s=0x55bd58976f00, done=0, tbc=1
2023-07-26T12:08:14.000Z 151291: : handle_udp_packet: 111.111: thrid=0x7f966bfff640: Amap = 0x7f9668a697b0, socket container=0x7f9668a697b0, local addr <removed>:3478, remote addr <removed>, s=0x55bd58976f00, done=0, tbc=1
2023-07-26T12:08:14.000Z 151291: : handle_udp_packet: 111.111: thrid=0x7f966bfff640: Amap = 0x7f9668a697b0, socket container=0x7f9668a697b0, local addr <removed>:3478, remote addr <removed>, s=0x55bd58976f00, done=0, tbc=1
 111.111: thrid=0x7f966bfff640: Amap = 0x7f9668a697b0, socket container=0x7f9668a697b0, local addr <removed>:3478, remote addr <removed>, s=0x55bd58976f00, done=0, tbc=1
2023-07-26T12:08:14.000Z 151291: : handle_udp_packet: 111.111: thrid=0x7f966bfff640: Amap = 0x7f9668a697b0, socket container=0x7f9668a697b0, local addr <removed>:3478, remote addr <removed>, s=0x55bd58976f00, done=0, tbc=1
2023-07-26T12:08:14.000Z 151291: : handle_udp_packet: 111.111: thrid=0x7f966bfff640: Amap = 0x7f9668a697b0, socket container=0x7f9668a697b0, local addr <removed>:3478, remote addr <removed>, s=0x55bd58976f00, done=0, tbc=1
2023-07-26T12:08:14.000Z 151291: : handle_udp_packet: 111.111: thrid=0x7f966bfff640: Amap = 0x7f9668a697b0, socket container=0x7f9668a697b0, local addr <removed>:3478, remote addr <removed>, s=0x55bd58976f00, done=0, tbc=1
2023-07-26T12:08:14.000Z 151291: : handle_udp_packet: 111.111: thrid=0x7f966bfff640: Amap = 0x7f9668a697b0, socket container=0x7f9668a697b0, local addr <removed>:3478, remote addr <removed>, s=0x55bd58976f00, done=0, tbc=1
2023-07-26T12:08:14.000Z 151291: : handle_udp_packet: 111.111: thrid=0x7f966bfff640: Amap = 0x7f9668a697b0, socket container=0x7f9668a697b0, local addr <removed>:3478, remote addr <removed>, s=0x55bd58976f00, done=0, tbc=1
2023-07-26T12:08:14.000Z 151291: : handle_udp_packet: 111.111: thrid=0x7f966bfff640: Amap = 0x7f9668a697b0, socket container=0x7f9668a697b0, local addr <removed>:3478, remote addr <removed>, s=0x55bd58976f00, done=0, tbc=1
2023-07-26T12:08:14.000Z 151292: : ERROR: handle_udp_packet: socket to be closed
2023-07-26T12:08:14.000Z 151292: : ERROR: handle_udp_packet: socket to be closed
2023-07-26T12:08:14.000Z 151292: : handle_udp_packet: 111.111: thrid=0x7f966bfff640: Amap = 0x7f9668a697b0, socket container=0x7f9668a697b0, local addr <removed>:3478, remote addr <removed>, s=0x55bd58976f00, done=0, tbc=1
2023-07-26T12:08:14.000Z 151292: : handle_udp_packet: 111.111: thrid=0x7f966bfff640: Amap = 0x7f9668a697b0, socket container=0x7f9668a697b0, local addr <removed>:3478, remote addr <removed>, s=0x55bd58976f00, done=0, tbc=1
2023-07-26T12:08:14.000Z udp send: Invalid argument
2023-07-26T12:08:14.000Z udp send: Invalid argument
2023-07-26T12:08:14.000Z udp send: Invalid argument
2023-07-26T12:08:14.000Z udp send: Invalid argument
2023-07-26T12:08:14.000Z udp send: Invalid argument
2023-07-26T12:08:14.000Z udp send: Invalid argument
              girishG 1 Reply Last reply
              0
              • W warg

                I just checked the error log of turn now (before it was nearly completely empty, likely due to reboot/kill of app) and it shows tons of errors (just a few of 35k lines):

                2023-07-26T12:08:13.000Z udp send: Invalid argument
2023-07-26T12:08:13.000Z udp send: Invalid argument
2023-07-26T12:08:13.000Z udp send: Invalid argument
2023-07-26T12:08:13.000Z udp send: Invalid argument
2023-07-26T12:08:13.000Z udp send: Invalid argument
2023-07-26T12:08:13.000Z udp send: Invalid argument
2023-07-26T12:08:13.000Z udp send: Invalid argument
2023-07-26T12:08:13.000Z udp send: Invalid argument
2023-07-26T12:08:13.000Z udp send: Invalid argument
2023-07-26T12:08:13.000Z udp send: Invalid argument
2023-07-26T12:08:13.000Z udp send: Invalid argument
2023-07-26T12:08:13.000Z udp send: Invalid argument
2023-07-26T12:08:13.000Z udp send: Invalid argument
2023-07-26T12:08:13.000Z udp send: Invalid argument
2023-07-26T12:08:13.000Z udp send: Invalid argument
2023-07-26T12:08:13.000Z udp send: Invalid argument
2023-07-26T12:08:13.000Z udp send: Invalid argument
2023-07-26T12:08:13.000Z udp send: Invalid argument
2023-07-26T12:08:13.000Z udp send: Invalid argument
2023-07-26T12:08:14.000Z 151291: : ERROR: handle_udp_packet: socket to be closed
2023-07-26T12:08:14.000Z 151291: : ERROR: handle_udp_packet: socket to be closed
2023-07-26T12:08:14.000Z 151291: : ERROR: handle_udp_packet: socket to be closed
2023-07-26T12:08:14.000Z 151291: : ERROR: handle_udp_packet: socket to be closed
2023-07-26T12:08:14.000Z 151291: : ERROR: handle_udp_packet: socket to be closed
2023-07-26T12:08:14.000Z 151291: : ERROR: handle_udp_packet: socket to be closed
2023-07-26T12:08:14.000Z 151291: : ERROR: handle_udp_packet: socket to be closed
2023-07-26T12:08:14.000Z 151291: : ERROR: handle_udp_packet: socket to be closed
2023-07-26T12:08:14.000Z 151291: : ERROR: handle_udp_packet: socket to be closed
2023-07-26T12:08:14.000Z 151291: : ERROR: handle_udp_packet: socket to be closed
2023-07-26T12:08:14.000Z 151291: : handle_udp_packet: 111.111: thrid=0x7f966bfff640: Amap = 0x7f9668a697b0, socket container=0x7f9668a697b0, local addr <removed>:3478, remote addr <removed>, s=0x55bd58976f00, done=0, tbc=1
2023-07-26T12:08:14.000Z 151291: : handle_udp_packet: 111.111: thrid=0x7f966bfff640: Amap = 0x7f9668a697b0, socket container=0x7f9668a697b0, local addr <removed>:3478, remote addr <removed>, s=0x55bd58976f00, done=0, tbc=1
2023-07-26T12:08:14.000Z 151291: : handle_udp_packet: 111.111: thrid=0x7f966bfff640: Amap = 0x7f9668a697b0, socket container=0x7f9668a697b0, local addr <removed>:3478, remote addr <removed>, s=0x55bd58976f00, done=0, tbc=1
 111.111: thrid=0x7f966bfff640: Amap = 0x7f9668a697b0, socket container=0x7f9668a697b0, local addr <removed>:3478, remote addr <removed>, s=0x55bd58976f00, done=0, tbc=1
2023-07-26T12:08:14.000Z 151291: : handle_udp_packet: 111.111: thrid=0x7f966bfff640: Amap = 0x7f9668a697b0, socket container=0x7f9668a697b0, local addr <removed>:3478, remote addr <removed>, s=0x55bd58976f00, done=0, tbc=1
2023-07-26T12:08:14.000Z 151291: : handle_udp_packet: 111.111: thrid=0x7f966bfff640: Amap = 0x7f9668a697b0, socket container=0x7f9668a697b0, local addr <removed>:3478, remote addr <removed>, s=0x55bd58976f00, done=0, tbc=1
2023-07-26T12:08:14.000Z 151291: : handle_udp_packet: 111.111: thrid=0x7f966bfff640: Amap = 0x7f9668a697b0, socket container=0x7f9668a697b0, local addr <removed>:3478, remote addr <removed>, s=0x55bd58976f00, done=0, tbc=1
2023-07-26T12:08:14.000Z 151291: : handle_udp_packet: 111.111: thrid=0x7f966bfff640: Amap = 0x7f9668a697b0, socket container=0x7f9668a697b0, local addr <removed>:3478, remote addr <removed>, s=0x55bd58976f00, done=0, tbc=1
2023-07-26T12:08:14.000Z 151291: : handle_udp_packet: 111.111: thrid=0x7f966bfff640: Amap = 0x7f9668a697b0, socket container=0x7f9668a697b0, local addr <removed>:3478, remote addr <removed>, s=0x55bd58976f00, done=0, tbc=1
2023-07-26T12:08:14.000Z 151291: : handle_udp_packet: 111.111: thrid=0x7f966bfff640: Amap = 0x7f9668a697b0, socket container=0x7f9668a697b0, local addr <removed>:3478, remote addr <removed>, s=0x55bd58976f00, done=0, tbc=1
2023-07-26T12:08:14.000Z 151292: : ERROR: handle_udp_packet: socket to be closed
2023-07-26T12:08:14.000Z 151292: : ERROR: handle_udp_packet: socket to be closed
2023-07-26T12:08:14.000Z 151292: : handle_udp_packet: 111.111: thrid=0x7f966bfff640: Amap = 0x7f9668a697b0, socket container=0x7f9668a697b0, local addr <removed>:3478, remote addr <removed>, s=0x55bd58976f00, done=0, tbc=1
2023-07-26T12:08:14.000Z 151292: : handle_udp_packet: 111.111: thrid=0x7f966bfff640: Amap = 0x7f9668a697b0, socket container=0x7f9668a697b0, local addr <removed>:3478, remote addr <removed>, s=0x55bd58976f00, done=0, tbc=1
2023-07-26T12:08:14.000Z udp send: Invalid argument
2023-07-26T12:08:14.000Z udp send: Invalid argument
2023-07-26T12:08:14.000Z udp send: Invalid argument
2023-07-26T12:08:14.000Z udp send: Invalid argument
2023-07-26T12:08:14.000Z udp send: Invalid argument
2023-07-26T12:08:14.000Z udp send: Invalid argument
                girishG Offline
                girishG Offline
                girish
                Staff
                wrote on last edited by
                #7

                @warg are you using the apps that use turn ?

                Tried searching for these error messages from coturn but I am not seeing getting any leads so far.

                1 Reply Last reply
                0
                • W Offline
                  W Offline
                  warg
                  wrote on last edited by warg
                  #8

                  I never heard of turn protocol/daemon before so I don't know if and which app uses it internally. Actively (= I'm aware of it): Nope. These apps are installed on my Cloudron v7.5.0:

                  • bookstack
                  • collabora
                  • lamp
                  • n8n
                  • nextcloud
                  • nocodb

                  These were recently removed:

                  • LibreTranslate
                  • Cloudron Build Service
                  1 Reply Last reply
                  0
                  • W Offline
                    W Offline
                    warg
                    wrote on last edited by
                    #9

                    I just rebooted the server to apply kernel updates from livepatching and I see errors like this from the startup (I enabled IPv6 on Cloudron short-term but I'm not sure it's relevant for this issue):

                    2023-07-27T16:16:09.000Z 0: : IO method (general relay thread): epoll (with changelist)
2023-07-27T16:16:09.000Z 0: : IO method (general relay thread): epoll (with changelist)
2023-07-27T16:16:09.000Z 0: : IO method (general relay thread): epoll (with changelist)
2023-07-27T16:16:09.000Z 0: : IO method (general relay thread): epoll (with changelist)
2023-07-27T16:16:09.000Z 0: : IO method (general relay thread): epoll (with changelist)
2023-07-27T16:16:09.000Z 0: : IO method (general relay thread): epoll (with changelist)
2023-07-27T16:16:09.000Z 0: : IO method (general relay thread): epoll (with changelist)
2023-07-27T16:16:09.000Z 0: : IO method (general relay thread): epoll (with changelist)
2023-07-27T16:16:09.000Z 0: : IO method (general relay thread): epoll (with changelist)
2023-07-27T16:16:09.000Z 0: : IO method (general relay thread): epoll (with changelist)
2023-07-27T16:16:09.000Z 0: : IO method (general relay thread): epoll (with changelist)
2023-07-27T16:16:09.000Z 0: : IO method (general relay thread): epoll (with changelist)
2023-07-27T16:16:09.000Z 0: : IO method (main listener thread): epoll (with changelist)
2023-07-27T16:16:09.000Z 0: : IO method (main listener thread): epoll (with changelist)
2023-07-27T16:16:09.000Z 0: : Listener address to use: 127.0.0.1
2023-07-27T16:16:09.000Z 0: : Listener address to use: 127.0.0.1
2023-07-27T16:16:09.000Z 0: : Listener address to use: 172.17.0.1
2023-07-27T16:16:09.000Z 0: : Listener address to use: 172.17.0.1
2023-07-27T16:16:09.000Z 0: : Listener address to use: 172.18.0.1
2023-07-27T16:16:09.000Z 0: : Listener address to use: 172.18.0.1
2023-07-27T16:16:09.000Z 0: : Listener address to use: <removed>
2023-07-27T16:16:09.000Z 0: : Listener address to use: <removed>
2023-07-27T16:16:09.000Z 0: : Listener address to use: <removed>
2023-07-27T16:16:09.000Z 0: : Listener address to use: <removed>
2023-07-27T16:16:09.000Z 0: : Listener address to use: ::1
2023-07-27T16:16:09.000Z 0: : Listener address to use: ::1
2023-07-27T16:16:09.000Z 0: : Listener address to use: <removed>
2023-07-27T16:16:09.000Z 0: : Listener address to use: <removed>
2023-07-27T16:16:09.000Z 0: : MongoDB is not supported
2023-07-27T16:16:09.000Z 0: : MongoDB is not supported
2023-07-27T16:16:09.000Z 0: : MySQL supported
2023-07-27T16:16:09.000Z 0: : MySQL supported
2023-07-27T16:16:09.000Z 0: : NO EXPLICIT LISTENER ADDRESS(ES) ARE CONFIGURED
2023-07-27T16:16:09.000Z 0: : NO EXPLICIT LISTENER ADDRESS(ES) ARE CONFIGURED
2023-07-27T16:16:09.000Z 0: : NO EXPLICIT RELAY ADDRESS(ES) ARE CONFIGURED
2023-07-27T16:16:09.000Z 0: : NO EXPLICIT RELAY ADDRESS(ES) ARE CONFIGURED
2023-07-27T16:16:09.000Z 0: : OpenSSL compile-time version: OpenSSL 3.0.2 15 Mar 2022 (0x30000020)
2023-07-27T16:16:09.000Z 0: : OpenSSL compile-time version: OpenSSL 3.0.2 15 Mar 2022 (0x30000020)
2023-07-27T16:16:09.000Z 0: : PostgreSQL supported
2023-07-27T16:16:09.000Z 0: : PostgreSQL supported
2023-07-27T16:16:09.000Z 0: : Redis supported
2023-07-27T16:16:09.000Z 0: : Redis supported
2023-07-27T16:16:09.000Z 0: : Relay address to use: 172.17.0.1
2023-07-27T16:16:09.000Z 0: : Relay address to use: 172.17.0.1
2023-07-27T16:16:09.000Z 0: : Relay address to use: 172.18.0.1
2023-07-27T16:16:09.000Z 0: : Relay address to use: 172.18.0.1
2023-07-27T16:16:09.000Z 0: : Relay address to use: <removed>
2023-07-27T16:16:09.000Z 0: : Relay address to use: <removed>
2023-07-27T16:16:09.000Z 0: : Relay address to use: <removed>
2023-07-27T16:16:09.000Z 0: : Relay address to use: <removed>
2023-07-27T16:16:09.000Z 0: : Relay address to use: <removed>
2023-07-27T16:16:09.000Z 0: : Relay address to use: <removed>
2023-07-27T16:16:09.000Z 0: : Relay ports initialization done
2023-07-27T16:16:09.000Z 0: : Relay ports initialization done
2023-07-27T16:16:09.000Z 0: : SQLite supported, default database location is /var/lib/turn/turndb
2023-07-27T16:16:09.000Z 0: : SQLite supported, default database location is /var/lib/turn/turndb
2023-07-27T16:16:09.000Z 0: : SSL23: Certificate file found: /run/turnserver/turn.cloudron.cert
2023-07-27T16:16:09.000Z 0: : SSL23: Certificate file found: /run/turnserver/turn.cloudron.cert
2023-07-27T16:16:09.000Z 0: : SSL23: Private key file found: /run/turnserver/turn.cloudron.key
2023-07-27T16:16:09.000Z 0: : SSL23: Private key file found: /run/turnserver/turn.cloudron.key
2023-07-27T16:16:09.000Z 0: : TLS cipher suite: ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384
2023-07-27T16:16:09.000Z 0: : TLS cipher suite: ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384
2023-07-27T16:16:09.000Z 0: : TLS supported
2023-07-27T16:16:09.000Z 0: : TLS supported
2023-07-27T16:16:09.000Z 0: : TLS1.2: Certificate file found: /run/turnserver/turn.cloudron.cert
2023-07-27T16:16:09.000Z 0: : TLS1.2: Certificate file found: /run/turnserver/turn.cloudron.cert
2023-07-27T16:16:09.000Z 0: : TLS1.2: Private key file found: /run/turnserver/turn.cloudron.key
2023-07-27T16:16:09.000Z 0: : TLS1.2: Private key file found: /run/turnserver/turn.cloudron.key
2023-07-27T16:16:09.000Z 0: : TURN/STUN ALPN supported
2023-07-27T16:16:09.000Z 0: : TURN/STUN ALPN supported
2023-07-27T16:16:09.000Z 0: : Third-party authorization (oAuth) supported
2023-07-27T16:16:09.000Z 0: : Third-party authorization (oAuth) supported
2023-07-27T16:16:09.000Z 0: : Total: 5 'real' addresses discovered
2023-07-27T16:16:09.000Z 0: : Total: 5 'real' addresses discovered
2023-07-27T16:16:09.000Z 0: : Total: 5 relay addresses discovered
2023-07-27T16:16:09.000Z 0: : Total: 5 relay addresses discovered
2023-07-27T16:16:09.000Z 0: : Trying to bind DTLS/UDP listener socket to addr [<removed>]:3478, again...
2023-07-27T16:16:09.000Z 0: : Trying to bind DTLS/UDP listener socket to addr [<removed>]:3478, again...
2023-07-27T16:16:09.000Z 0: : Trying to bind TLS/TCP listener socket to addr [<removed>]:3478, again...
2023-07-27T16:16:09.000Z 0: : Trying to bind TLS/TCP listener socket to addr [<removed>]:3478, again...
2023-07-27T16:16:09.000Z 0: : Trying to bind TLS/TCP listener socket to addr [<removed>]:3478, again...
2023-07-27T16:16:09.000Z 0: : Trying to bind TLS/TCP listener socket to addr [<removed>]:3478, again...
2023-07-27T16:16:09.000Z 0: : Trying to bind TLS/TCP listener socket to addr [<removed>]:3478, again...
2023-07-27T16:16:09.000Z 0: : Trying to bind TLS/TCP listener socket to addr [<removed>]:3478, again...
2023-07-27T16:16:09.000Z 0: : Trying to bind TLS/TCP listener socket to addr [<removed>]:3478, again...
2023-07-27T16:16:09.000Z 0: : Trying to bind TLS/TCP listener socket to addr [<removed>]:3478, again...
2023-07-27T16:16:09.000Z 0: : Trying to bind TLS/TCP listener socket to addr [<removed>]:3478, again...
2023-07-27T16:16:09.000Z 0: : Trying to bind TLS/TCP listener socket to addr [<removed>]:3478, again...
2023-07-27T16:16:09.000Z 0: : Trying to bind TLS/TCP listener socket to addr [<removed>]:3478, again...
2023-07-27T16:16:09.000Z 0: : Trying to bind TLS/TCP listener socket to addr [<removed>]:3478, again...
2023-07-27T16:16:09.000Z 0: : Trying to bind fd 175 to <[<removed>]:3478>: errno=99
2023-07-27T16:16:09.000Z 0: : Trying to bind fd 175 to <[<removed>]:3478>: errno=99
2023-07-27T16:16:09.000Z 0: : Trying to bind fd 249 to <[<removed>]:3478>: errno=99
2023-07-27T16:16:09.000Z 0: : Trying to bind fd 249 to <[<removed>]:3478>: errno=99
2023-07-27T16:16:09.000Z 0: : Trying to bind fd 309 to <[<removed>]:3478>: errno=99
2023-07-27T16:16:09.000Z 0: : Trying to bind fd 309 to <[<removed>]:3478>: errno=99
2023-07-27T16:16:09.000Z 0: : Trying to bind fd 363 to <[<removed>]:3478>: errno=99
2023-07-27T16:16:09.000Z 0: : Trying to bind fd 363 to <[<removed>]:3478>: errno=99
2023-07-27T16:16:09.000Z 0: : Trying to bind fd 381 to <[<removed>]:3478>: errno=99
2023-07-27T16:16:09.000Z 0: : Trying to bind fd 381 to <[<removed>]:3478>: errno=99
2023-07-27T16:16:09.000Z 0: : Trying to bind fd 458 to <[<removed>]:3478>: errno=99
2023-07-27T16:16:09.000Z 0: : Trying to bind fd 458 to <[<removed>]:3478>: errno=99
2023-07-27T16:16:09.000Z 0: : Trying to bind fd 469 to <[<removed>]:3478>: errno=99
2023-07-27T16:16:09.000Z 0: : Trying to bind fd 469 to <[<removed>]:3478>: errno=99
2023-07-27T16:16:09.000Z 0: : Wait for relay ports initialization...
2023-07-27T16:16:09.000Z 0: : Wait for relay ports initialization...
2023-07-27T16:16:09.000Z 0: : pid file created: /run/turnserver/turnserver.pid
2023-07-27T16:16:09.000Z 0: : pid file created: /run/turnserver/turnserver.pid
2023-07-27T16:16:09.000Z 0: : turn server id=0 created
2023-07-27T16:16:09.000Z 0: : turn server id=0 created
2023-07-27T16:16:09.000Z 0: : turn server id=1 created
2023-07-27T16:16:09.000Z 0: : turn server id=1 created
2023-07-27T16:16:09.000Z 0: : turn server id=2 created
2023-07-27T16:16:09.000Z 0: : turn server id=2 created
2023-07-27T16:16:09.000Z 0: : turn server id=3 created
2023-07-27T16:16:09.000Z 0: : turn server id=3 created
2023-07-27T16:16:09.000Z 0: : turn server id=4 created
2023-07-27T16:16:09.000Z 0: : turn server id=4 created
2023-07-27T16:16:09.000Z 0: : turn server id=5 created
2023-07-27T16:16:09.000Z 0: : turn server id=5 created
2023-07-27T16:16:09.000Z ==== Show him the instruments, Practical Frost: ====
2023-07-27T16:16:09.000Z ==== Show him the instruments, Practical Frost: ====
2023-07-27T16:16:09.000Z =====================================================
2023-07-27T16:16:09.000Z =====================================================
2023-07-27T16:16:09.000Z => Ensure permissions
2023-07-27T16:16:09.000Z => Generating turnserver.conf
2023-07-27T16:16:09.000Z => Starting coturn
2023-07-27T16:16:09.000Z Be aware that you could not mix the username/password and the shared secret based auth methods.
2023-07-27T16:16:09.000Z Be aware that you could not mix the username/password and the shared secret based auth methods.
2023-07-27T16:16:09.000Z CONFIGURATION ALERT: You specified --lt-cred-mech and --use-auth-secret in the same time.
2023-07-27T16:16:09.000Z CONFIGURATION ALERT: You specified --lt-cred-mech and --use-auth-secret in the same time.
2023-07-27T16:16:09.000Z Cannot bind local socket to addr: Cannot assign requested address
2023-07-27T16:16:09.000Z Cannot bind local socket to addr: Cannot assign requested address
2023-07-27T16:16:09.000Z Cannot bind local socket to addr: Cannot assign requested address
2023-07-27T16:16:09.000Z Cannot bind local socket to addr: Cannot assign requested address
2023-07-27T16:16:09.000Z Cannot bind local socket to addr: Cannot assign requested address
2023-07-27T16:16:09.000Z Cannot bind local socket to addr: Cannot assign requested address
2023-07-27T16:16:09.000Z Cannot bind local socket to addr: Invalid argument
2023-07-27T16:16:09.000Z Due to the open files/sockets limitation,
2023-07-27T16:16:09.000Z Due to the open files/sockets limitation,
2023-07-27T16:16:09.000Z Max number of open files/sockets allowed for this process: 1048576
2023-07-27T16:16:09.000Z Max number of open files/sockets allowed for this process: 1048576
2023-07-27T16:16:09.000Z RFC 3489/5389/5766/5780/6062/6156 STUN/TURN Server
2023-07-27T16:16:09.000Z RFC 3489/5389/5766/5780/6062/6156 STUN/TURN Server
2023-07-27T16:16:09.000Z Shared secret overrides username/password based auth method. Check your configuration!
2023-07-27T16:16:09.000Z Shared secret overrides username/password based auth method. Check your configuration!
2023-07-27T16:16:09.000Z Version Coturn-4.5.2 'dan Eider'
2023-07-27T16:16:09.000Z Version Coturn-4.5.2 'dan Eider'
2023-07-27T16:16:09.000Z bind: Cannot assign requested address
2023-07-27T16:16:09.000Z bind: Cannot assign requested address
2023-07-27T16:16:09.000Z bind: Cannot assign requested address
2023-07-27T16:16:09.000Z bind: Cannot assign requested address
2023-07-27T16:16:09.000Z bind: Cannot assign requested address
2023-07-27T16:16:09.000Z bind: Cannot assign requested address
2023-07-27T16:16:09.000Z bind: Cannot assign requested address
2023-07-27T16:16:09.000Z max supported number of TURN Sessions possible is: 524000 (approximately)
2023-07-27T16:16:09.000Z max supported number of TURN Sessions possible is: 524000 (approximately)
2023-07-27T16:16:10.000Z 0: : IO method (auth thread): epoll (with changelist)
2023-07-27T16:16:10.000Z 0: : IO method (auth thread): epoll (with changelist)
2023-07-27T16:16:10.000Z 0: : Total General servers: 6
2023-07-27T16:16:10.000Z 0: : Total General servers: 6
2023-07-27T16:16:10.000Z 1: : IO method (admin thread): epoll (with changelist)
2023-07-27T16:16:10.000Z 1: : IO method (admin thread): epoll (with changelist)
2023-07-27T16:16:10.000Z 1: : IO method (auth thread): epoll (with changelist)
2023-07-27T16:16:10.000Z 1: : IO method (auth thread): epoll (with changelist)
2023-07-27T16:16:10.000Z 1: : IO method (auth thread): epoll (with changelist)
2023-07-27T16:16:10.000Z 1: : IO method (auth thread): epoll (with changelist)
2023-07-27T16:16:10.000Z 1: : New GID: turnserver(108)
2023-07-27T16:16:10.000Z 1: : New GID: turnserver(108)
2023-07-27T16:16:10.000Z 1: : New UID: turnserver(106)
2023-07-27T16:16:10.000Z 1: : New UID: turnserver(106)
2023-07-27T16:16:10.000Z 1: : SQLite DB connection success: /var/lib/turn/turndb
2023-07-27T16:16:10.000Z 1: : SQLite DB connection success: /var/lib/turn/turndb

                    Looks there's some IPv6 IP address binding issue and a max file or socket limit reached error.

                    1 Reply Last reply
                    0
                    • marcusquinnM Offline
                      marcusquinnM Offline
                      marcusquinn
                      wrote on last edited by marcusquinn
                      #10

                      Interesting. I had this recently, too. Memory allowance seems to be about 200% more than used. Seems to have righted itself, so just noting in case relevant and others noticed the same recently, too. Only happened once for as long as I can remember, so perhaps nothing, but noting just because occasional things can be hard to understand without evidence and reproducibility.

                      Web Design https://www.evergreen.je
                      Development https://brandlight.org
                      Life https://marcusquinn.com

                      1 Reply Last reply
                      0
                      • W Offline
                        W Offline
                        warg
                        wrote on last edited by warg
                        #11

                        The critical errors with invalid arguments doesn't come up anymore. I think this came up from some Ubuntu Livepatching being done + missing reboot afterwards. Didn't occur the last days and for extra safety I just updated all packages on the server now. Nothing broke (although it's not suggested to do that) and I will see if issue occurs again.

                        1 Reply Last reply
                        1
                        • nebulonN Offline
                          nebulonN Offline
                          nebulon
                          Staff
                          wrote on last edited by
                          #12

                          hopefully critical components like docker were not updated then, since this will eventually break. Make sure to have full backups.

                          1 Reply Last reply
                          1
                          • W Offline
                            W Offline
                            warg
                            wrote on last edited by
                            #13

                            Thanks for the hint. I accepted that it might break. So far there were just OS-related components updated and DB-stuff. At the moment I saw no critical errors in the apps and they do work. Cloudron works fine as well at the moment. I also restarted the server to see if somethings breaks on full reload. So far, everything is fine and docker-related stuff wasn't updated.

                            1 Reply Last reply
                            0
                            • W Offline
                              W Offline
                              warg
                              wrote on last edited by warg
                              #14

                              Nope, issue was not related to livepatching it seems. Issue just occured out of nowhere again:

                              grafik.png

                              Could this be related to malicious clients, e. g. erros created by bots that try turn-specific exploits?

                              girishG 1 Reply Last reply
                              0
                              • W warg

                                Nope, issue was not related to livepatching it seems. Issue just occured out of nowhere again:

                                grafik.png

                                Could this be related to malicious clients, e. g. erros created by bots that try turn-specific exploits?

                                girishG Offline
                                girishG Offline
                                girish
                                Staff
                                wrote on last edited by
                                #15

                                @warg said in The turn service ran out of memory:

                                Could this be related to malicious clients, e. g. erros created by bots that try turn-specific exploits?

                                It does seem to be like that.

                                Is anyone else seeing such messages in the turn service logs?

                                1 Reply Last reply
                                0
                                Reply
                                • Reply as topic
                                Log in to reply
                                • Oldest to Newest
                                • Newest to Oldest
                                • Most Votes

                                  • Login
                                  • Don't have an account? Register
                                  • Login or register to search.
                                  • First post
                                    Last post
                                  0
                                  • Categories
                                  • Recent
                                  • Tags
                                  • Popular
                                  • Bookmarks
                                  • Search