Long startup and hourly crash

[robi]

New install on demo server..

Nov 12 14:28:50=> Healtheck error: Error: connect ECONNREFUSED 172.18.16.18:80
Nov 12 14:28:58
Nov 12 14:28:58██  ██  █████  ██████  ███  ██ ██ ███  ██  ██████
Nov 12 14:28:58██  ██ ██   ██ ██   ██ ████  ██ ██ ████  ██ ██
Nov 12 14:28:58██  █  ██ ███████ ██████  ██ ██  ██ ██ ██ ██  ██ ██  ███
Nov 12 14:28:58██ ███ ██ ██   ██ ██   ██ ██  ██ ██ ██ ██  ██ ██ ██  ██
Nov 12 14:28:58███ ███  ██  ██ ██  ██ ██   ████ ██ ██   ████  ██████
Nov 12 14:28:58
Nov 12 14:28:58******************************************************************************
Nov 12 14:28:58Your database has a single connection, and single connections were
Nov 12 14:28:58deprecated in GitLab 15.9 https://docs.gitlab.com/ee/update/deprecations.html#single-database-connection-is-deprecated.
Nov 12 14:28:58
Nov 12 14:28:58Please add a :ci section to your database, following these instructions:
Nov 12 14:28:58https://docs.gitlab.com/ee/install/installation.html#configure-gitlab-db-settings.
Nov 12 14:28:58******************************************************************************

Seems the package needs some updates..

Takes forever to start up.. and if left alone uses up all 3GB of RAM and crashes, repeating the process.

Setting the HOME=/app/data env may also be helpful for other errors about /home/git in the logs.

Nov 12 18:29:21time="2023-11-13T02:29:21.126Z" level=info msg="[transport] [server-transport 0xc003da44e0] Closing: EOF" pid=227 system=system
Nov 12 18:29:212023-11-13 02:29:21,127 INFO exited: sidekiq (terminated by SIGKILL; not expected)
Nov 12 18:29:21time="2023-11-13T02:29:21.127Z" level=info msg="[transport] [server-transport 0xc003da44e0] loopyWriter exiting with error: transport closed by client" pid=227 system=system
Nov 12 18:29:212023-11-13 02:29:21,134 INFO spawned: 'sidekiq' with pid 1106
Nov 12 18:29:21`/home/git` is not writable.

[girish]

The instance on the demo is not there anymore but I can remove the db deprecation warning. I am guessing it's dying with SIGKILL because of memory constraints.

[girish]

Published new package for the deprecation warning.

[robi]

Crash still happening on new demo instance.
2023-11-13T20:08:32.000Z 2023-11-13 20:08:32,435 INFO exited: sidekiq (terminated by SIGKILL; not expected)

I have log zip if needed.

[girish]

It's just running out of memory:

[312152.970350] puma srv thread invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
[312152.970356] CPU: 0 PID: 1635554 Comm: puma srv thread Not tainted 5.4.0-131-generic #147-Ubuntu
[312152.970357] Hardware name: Linode Compute Instance/Standard PC (Q35 + ICH9, 2009), BIOS Not Specified 
[312152.970358] Call Trace:
[312152.970367]  dump_stack+0x6d/0x8b
[312152.970371]  dump_header+0x4f/0x1eb
[312152.970374]  oom_kill_process.cold+0xb/0x10
[312152.970377]  out_of_memory+0x1cf/0x500
[312152.970379]  mem_cgroup_out_of_memory+0xbd/0xe0
[312152.970381]  try_charge+0x77c/0x810
[312152.970383]  mem_cgroup_try_charge+0x71/0x190
[312152.970386]  __add_to_page_cache_locked+0x2ff/0x3f0
[312152.970388]  ? scan_shadow_nodes+0x30/0x30
[312152.970390]  add_to_page_cache_lru+0x4d/0xd0
[312152.970391]  pagecache_get_page+0x101/0x300
[312152.970393]  filemap_fault+0x6b2/0xa50
[312152.970395]  ? unlock_page_memcg+0x12/0x20
[312152.970396]  ? page_add_file_rmap+0xff/0x1a0
[312152.970398]  ? filemap_map_pages+0x24c/0x380
[312152.970401]  ext4_filemap_fault+0x32/0x50
[312152.970403]  __do_fault+0x3c/0x170
[312152.970404]  do_fault+0x24b/0x640
[312152.970406]  __handle_mm_fault+0x4c5/0x7a0
[312152.970407]  handle_mm_fault+0xca/0x200
[312152.970410]  do_user_addr_fault+0x1f9/0x450
[312152.970412]  __do_page_fault+0x58/0x90
[312152.970415]  ? exit_to_usermode_loop+0x106/0x160
[312152.970416]  do_page_fault+0x2c/0xe0
[312152.970418]  do_async_page_fault+0x39/0x70
[312152.970420]  async_page_fault+0x34/0x40
[312152.970422] RIP: 0033:0x7f00b0d3dfc6
[312152.970429] Code: Bad RIP value.
[312152.970430] RSP: 002b:00007f00aa6c6fc0 EFLAGS: 00010206
[312152.970432] RAX: 0000560213c4c280 RBX: 00007f00aa7c7af8 RCX: 00007f00b0b39520
[312152.970433] RDX: 0000000000000000 RSI: 0000000000000008 RDI: 00007f00aa6c6ed0
[312152.970433] RBP: 00005602204e6da0 R08: 0000000000000000 R09: 00007f00aa6c7d59
[312152.970434] R10: 00005601f82fc820 R11: 0000000000000001 R12: 00007f00aa7c7af8
[312152.970435] R13: 00005601f8bede00 R14: 0000560213c4c280 R15: 00007f00aa7c7af8
[312152.970437] memory: usage 2516992kB, limit 2516992kB, failcnt 254345
[312152.970438] memory+swap: usage 3145648kB, limit 3145728kB, failcnt 985908
[312152.970439] kmem: usage 26464kB, limit 9007199254740988kB, failcnt 0
[312152.970439] Memory cgroup stats for /docker/4570286f36c91ba31464f017556c26c70c539ee0abcc81fbaf1d7689d7615f0a:
[312152.970452] anon 2550063104
                file 249856
                kernel_stack 1363968
                slab 13627392
                sock 0
                shmem 0
                file_mapped 270336
                file_dirty 0
                file_writeback 405504
                anon_thp 0
                inactive_anon 509853696
                active_anon 2039803904
                inactive_file 0
                active_file 0
                unevictable 0
                slab_reclaimable 7442432
                slab_unreclaimable 6184960
                pgfault 2674155
                pgmajfault 228294
                workingset_refault 2706561
                workingset_activate 192984
                workingset_nodereclaim 0
                pgrefill 2434699
                pgscan 8689699
                pgsteal 3128433
                pgactivate 1466256
                pgdeactivate 2173314
                pglazyfree 0
                pglazyfreed 0
                thp_fault_alloc 0
                thp_collapse_alloc 0
[312152.970453] Tasks state (memory values in pages):
[312152.970453] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[312152.970458] [1634573]     0 1634573     8558     3239   106496     3660             0 supervisord
[312152.970460] [1635332]  1001 1635332   568756    42145   851968    24321             0 gitaly
[312152.970463] [1635333]  1001 1635333   300975    48774  1892352   158917             0 ruby3.0
[312152.970464] [1635339]  1001 1635339   501544     9685   282624      571             0 gitlab-workhors
[312152.970466] [1635341]     0 1635341    14691     1777   106496      801             0 nginx
[312152.970468] [1635342]     0 1635342     3855     1551    65536      407             0 sshd
[312152.970470] [1635356]    33 1635356    14691      448   102400      840             0 nginx
[312152.970472] [1635357]    33 1635357    14691      455   102400      838             0 nginx
[312152.970474] [1635358]    33 1635358    14691      424   102400      838             0 nginx
[312152.970475] [1635359]    33 1635359    14691       68   102400      853             0 nginx
[312152.970476] [1635360]    33 1635360    14691      220   102400      881             0 nginx
[312152.970478] [1635361]    33 1635361    14691      220   102400      881             0 nginx
[312152.970480] [1635549]  1001 1635549   403233   131047  2060288    92264             0 ruby3.0
[312152.970482] [1635559]  1001 1635559   403233   137637  2060288    85994             0 ruby3.0
[312152.970484] [1646271]  1001 1646271   586234   232443  2342912        0             0 ruby3.0
[312152.970486] [1646525]  1001 1646525   402405   133212  1990656    83169             0 ruby3.0
[312152.970487] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=4570286f36c91ba31464f017556c26c70c539ee0abcc81fbaf1d7689d7615f0a,mems_allowed=0,oom_memcg=/docker/4570286f36c91ba31464f017556c26c70c539ee0abcc81fbaf1d7689d7615f0a,task_memcg=/docker/4570286f36c91ba31464f017556c26c70c539ee0abcc81fbaf1d7689d7615f0a,task=ruby3.0,pid=1646271,uid=1001
[312152.970588] Memory cgroup out of memory: Killed process 1646271 (ruby3.0) total-vm:2344936kB, anon-rss:917944kB, file-rss:11828kB, shmem-rss:0kB, UID:1001 pgtables:2288kB oom_score_adj:0

[girish]

I bumped the memory to 4GB let's see

[girish]

4 cores and 4GB is the bare minimum it seems - https://docs.gitlab.com/ee/install/requirements.html#memory

[robi]

It's a beast, hence the req to bump the package cgroup mem threshold by default. Doesn't provide a good experience for new installs

[girish]

@robi right, it's much better with 4GB. I will publish a new package with the update memory requirements.

[robi]

It's better now, using 3250MB of RAM.