Long startup and hourly crash
GitLab
10
Posts
2
Posters
1.5k
Views
2
Watching
-
New install on demo server..
Nov 12 14:28:50=> Healtheck error: Error: connect ECONNREFUSED 172.18.16.18:80 Nov 12 14:28:58 Nov 12 14:28:58██ ██ █████ ██████ ███ ██ ██ ███ ██ ██████ Nov 12 14:28:58██ ██ ██ ██ ██ ██ ████ ██ ██ ████ ██ ██ Nov 12 14:28:58██ █ ██ ███████ ██████ ██ ██ ██ ██ ██ ██ ██ ██ ███ Nov 12 14:28:58██ ███ ██ ██ ██ ██ ██ ██ ██ ██ ██ ██ ██ ██ ██ ██ Nov 12 14:28:58███ ███ ██ ██ ██ ██ ██ ████ ██ ██ ████ ██████ Nov 12 14:28:58 Nov 12 14:28:58****************************************************************************** Nov 12 14:28:58Your database has a single connection, and single connections were Nov 12 14:28:58deprecated in GitLab 15.9 https://docs.gitlab.com/ee/update/deprecations.html#single-database-connection-is-deprecated. Nov 12 14:28:58 Nov 12 14:28:58Please add a :ci section to your database, following these instructions: Nov 12 14:28:58https://docs.gitlab.com/ee/install/installation.html#configure-gitlab-db-settings. Nov 12 14:28:58******************************************************************************Seems the package needs some updates..
Takes forever to start up.. and if left alone uses up all 3GB of RAM and crashes, repeating the process.
Setting the HOME=/app/data env may also be helpful for other errors about /home/git in the logs.
Nov 12 18:29:21time="2023-11-13T02:29:21.126Z" level=info msg="[transport] [server-transport 0xc003da44e0] Closing: EOF" pid=227 system=system Nov 12 18:29:212023-11-13 02:29:21,127 INFO exited: sidekiq (terminated by SIGKILL; not expected) Nov 12 18:29:21time="2023-11-13T02:29:21.127Z" level=info msg="[transport] [server-transport 0xc003da44e0] loopyWriter exiting with error: transport closed by client" pid=227 system=system Nov 12 18:29:212023-11-13 02:29:21,134 INFO spawned: 'sidekiq' with pid 1106 Nov 12 18:29:21`/home/git` is not writable. -
The instance on the demo is not there anymore but I can remove the db deprecation warning. I am guessing it's dying with SIGKILL because of memory constraints.
-
Published new package for the deprecation warning.
-
Crash still happening on new demo instance.
2023-11-13T20:08:32.000Z 2023-11-13 20:08:32,435 INFO exited: sidekiq (terminated by SIGKILL; not expected)I have log zip if needed.
-
It's just running out of memory:
[312152.970350] puma srv thread invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0 [312152.970356] CPU: 0 PID: 1635554 Comm: puma srv thread Not tainted 5.4.0-131-generic #147-Ubuntu [312152.970357] Hardware name: Linode Compute Instance/Standard PC (Q35 + ICH9, 2009), BIOS Not Specified [312152.970358] Call Trace: [312152.970367] dump_stack+0x6d/0x8b [312152.970371] dump_header+0x4f/0x1eb [312152.970374] oom_kill_process.cold+0xb/0x10 [312152.970377] out_of_memory+0x1cf/0x500 [312152.970379] mem_cgroup_out_of_memory+0xbd/0xe0 [312152.970381] try_charge+0x77c/0x810 [312152.970383] mem_cgroup_try_charge+0x71/0x190 [312152.970386] __add_to_page_cache_locked+0x2ff/0x3f0 [312152.970388] ? scan_shadow_nodes+0x30/0x30 [312152.970390] add_to_page_cache_lru+0x4d/0xd0 [312152.970391] pagecache_get_page+0x101/0x300 [312152.970393] filemap_fault+0x6b2/0xa50 [312152.970395] ? unlock_page_memcg+0x12/0x20 [312152.970396] ? page_add_file_rmap+0xff/0x1a0 [312152.970398] ? filemap_map_pages+0x24c/0x380 [312152.970401] ext4_filemap_fault+0x32/0x50 [312152.970403] __do_fault+0x3c/0x170 [312152.970404] do_fault+0x24b/0x640 [312152.970406] __handle_mm_fault+0x4c5/0x7a0 [312152.970407] handle_mm_fault+0xca/0x200 [312152.970410] do_user_addr_fault+0x1f9/0x450 [312152.970412] __do_page_fault+0x58/0x90 [312152.970415] ? exit_to_usermode_loop+0x106/0x160 [312152.970416] do_page_fault+0x2c/0xe0 [312152.970418] do_async_page_fault+0x39/0x70 [312152.970420] async_page_fault+0x34/0x40 [312152.970422] RIP: 0033:0x7f00b0d3dfc6 [312152.970429] Code: Bad RIP value. [312152.970430] RSP: 002b:00007f00aa6c6fc0 EFLAGS: 00010206 [312152.970432] RAX: 0000560213c4c280 RBX: 00007f00aa7c7af8 RCX: 00007f00b0b39520 [312152.970433] RDX: 0000000000000000 RSI: 0000000000000008 RDI: 00007f00aa6c6ed0 [312152.970433] RBP: 00005602204e6da0 R08: 0000000000000000 R09: 00007f00aa6c7d59 [312152.970434] R10: 00005601f82fc820 R11: 0000000000000001 R12: 00007f00aa7c7af8 [312152.970435] R13: 00005601f8bede00 R14: 0000560213c4c280 R15: 00007f00aa7c7af8 [312152.970437] memory: usage 2516992kB, limit 2516992kB, failcnt 254345 [312152.970438] memory+swap: usage 3145648kB, limit 3145728kB, failcnt 985908 [312152.970439] kmem: usage 26464kB, limit 9007199254740988kB, failcnt 0 [312152.970439] Memory cgroup stats for /docker/4570286f36c91ba31464f017556c26c70c539ee0abcc81fbaf1d7689d7615f0a: [312152.970452] anon 2550063104 file 249856 kernel_stack 1363968 slab 13627392 sock 0 shmem 0 file_mapped 270336 file_dirty 0 file_writeback 405504 anon_thp 0 inactive_anon 509853696 active_anon 2039803904 inactive_file 0 active_file 0 unevictable 0 slab_reclaimable 7442432 slab_unreclaimable 6184960 pgfault 2674155 pgmajfault 228294 workingset_refault 2706561 workingset_activate 192984 workingset_nodereclaim 0 pgrefill 2434699 pgscan 8689699 pgsteal 3128433 pgactivate 1466256 pgdeactivate 2173314 pglazyfree 0 pglazyfreed 0 thp_fault_alloc 0 thp_collapse_alloc 0 [312152.970453] Tasks state (memory values in pages): [312152.970453] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name [312152.970458] [1634573] 0 1634573 8558 3239 106496 3660 0 supervisord [312152.970460] [1635332] 1001 1635332 568756 42145 851968 24321 0 gitaly [312152.970463] [1635333] 1001 1635333 300975 48774 1892352 158917 0 ruby3.0 [312152.970464] [1635339] 1001 1635339 501544 9685 282624 571 0 gitlab-workhors [312152.970466] [1635341] 0 1635341 14691 1777 106496 801 0 nginx [312152.970468] [1635342] 0 1635342 3855 1551 65536 407 0 sshd [312152.970470] [1635356] 33 1635356 14691 448 102400 840 0 nginx [312152.970472] [1635357] 33 1635357 14691 455 102400 838 0 nginx [312152.970474] [1635358] 33 1635358 14691 424 102400 838 0 nginx [312152.970475] [1635359] 33 1635359 14691 68 102400 853 0 nginx [312152.970476] [1635360] 33 1635360 14691 220 102400 881 0 nginx [312152.970478] [1635361] 33 1635361 14691 220 102400 881 0 nginx [312152.970480] [1635549] 1001 1635549 403233 131047 2060288 92264 0 ruby3.0 [312152.970482] [1635559] 1001 1635559 403233 137637 2060288 85994 0 ruby3.0 [312152.970484] [1646271] 1001 1646271 586234 232443 2342912 0 0 ruby3.0 [312152.970486] [1646525] 1001 1646525 402405 133212 1990656 83169 0 ruby3.0 [312152.970487] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=4570286f36c91ba31464f017556c26c70c539ee0abcc81fbaf1d7689d7615f0a,mems_allowed=0,oom_memcg=/docker/4570286f36c91ba31464f017556c26c70c539ee0abcc81fbaf1d7689d7615f0a,task_memcg=/docker/4570286f36c91ba31464f017556c26c70c539ee0abcc81fbaf1d7689d7615f0a,task=ruby3.0,pid=1646271,uid=1001 [312152.970588] Memory cgroup out of memory: Killed process 1646271 (ruby3.0) total-vm:2344936kB, anon-rss:917944kB, file-rss:11828kB, shmem-rss:0kB, UID:1001 pgtables:2288kB oom_score_adj:0 -
I bumped the memory to 4GB let's see
-
4 cores and 4GB is the bare minimum it seems - https://docs.gitlab.com/ee/install/requirements.html#memory
-
@robi right, it's much better with 4GB. I will publish a new package with the update memory requirements.
