Long startup and hourly crash
-
New install on demo server..
Nov 12 14:28:50=> Healtheck error: Error: connect ECONNREFUSED 172.18.16.18:80 Nov 12 14:28:58 Nov 12 14:28:58██ ██ █████ ██████ ███ ██ ██ ███ ██ ██████ Nov 12 14:28:58██ ██ ██ ██ ██ ██ ████ ██ ██ ████ ██ ██ Nov 12 14:28:58██ █ ██ ███████ ██████ ██ ██ ██ ██ ██ ██ ██ ██ ███ Nov 12 14:28:58██ ███ ██ ██ ██ ██ ██ ██ ██ ██ ██ ██ ██ ██ ██ ██ Nov 12 14:28:58███ ███ ██ ██ ██ ██ ██ ████ ██ ██ ████ ██████ Nov 12 14:28:58 Nov 12 14:28:58****************************************************************************** Nov 12 14:28:58Your database has a single connection, and single connections were Nov 12 14:28:58deprecated in GitLab 15.9 https://docs.gitlab.com/ee/update/deprecations.html#single-database-connection-is-deprecated. Nov 12 14:28:58 Nov 12 14:28:58Please add a :ci section to your database, following these instructions: Nov 12 14:28:58https://docs.gitlab.com/ee/install/installation.html#configure-gitlab-db-settings. Nov 12 14:28:58******************************************************************************Seems the package needs some updates..
Takes forever to start up.. and if left alone uses up all 3GB of RAM and crashes, repeating the process.
Setting the HOME=/app/data env may also be helpful for other errors about /home/git in the logs.
Nov 12 18:29:21time="2023-11-13T02:29:21.126Z" level=info msg="[transport] [server-transport 0xc003da44e0] Closing: EOF" pid=227 system=system Nov 12 18:29:212023-11-13 02:29:21,127 INFO exited: sidekiq (terminated by SIGKILL; not expected) Nov 12 18:29:21time="2023-11-13T02:29:21.127Z" level=info msg="[transport] [server-transport 0xc003da44e0] loopyWriter exiting with error: transport closed by client" pid=227 system=system Nov 12 18:29:212023-11-13 02:29:21,134 INFO spawned: 'sidekiq' with pid 1106 Nov 12 18:29:21`/home/git` is not writable. -
Crash still happening on new demo instance.
2023-11-13T20:08:32.000Z 2023-11-13 20:08:32,435 INFO exited: sidekiq (terminated by SIGKILL; not expected)I have log zip if needed.
-
It's just running out of memory:
[312152.970350] puma srv thread invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0 [312152.970356] CPU: 0 PID: 1635554 Comm: puma srv thread Not tainted 5.4.0-131-generic #147-Ubuntu [312152.970357] Hardware name: Linode Compute Instance/Standard PC (Q35 + ICH9, 2009), BIOS Not Specified [312152.970358] Call Trace: [312152.970367] dump_stack+0x6d/0x8b [312152.970371] dump_header+0x4f/0x1eb [312152.970374] oom_kill_process.cold+0xb/0x10 [312152.970377] out_of_memory+0x1cf/0x500 [312152.970379] mem_cgroup_out_of_memory+0xbd/0xe0 [312152.970381] try_charge+0x77c/0x810 [312152.970383] mem_cgroup_try_charge+0x71/0x190 [312152.970386] __add_to_page_cache_locked+0x2ff/0x3f0 [312152.970388] ? scan_shadow_nodes+0x30/0x30 [312152.970390] add_to_page_cache_lru+0x4d/0xd0 [312152.970391] pagecache_get_page+0x101/0x300 [312152.970393] filemap_fault+0x6b2/0xa50 [312152.970395] ? unlock_page_memcg+0x12/0x20 [312152.970396] ? page_add_file_rmap+0xff/0x1a0 [312152.970398] ? filemap_map_pages+0x24c/0x380 [312152.970401] ext4_filemap_fault+0x32/0x50 [312152.970403] __do_fault+0x3c/0x170 [312152.970404] do_fault+0x24b/0x640 [312152.970406] __handle_mm_fault+0x4c5/0x7a0 [312152.970407] handle_mm_fault+0xca/0x200 [312152.970410] do_user_addr_fault+0x1f9/0x450 [312152.970412] __do_page_fault+0x58/0x90 [312152.970415] ? exit_to_usermode_loop+0x106/0x160 [312152.970416] do_page_fault+0x2c/0xe0 [312152.970418] do_async_page_fault+0x39/0x70 [312152.970420] async_page_fault+0x34/0x40 [312152.970422] RIP: 0033:0x7f00b0d3dfc6 [312152.970429] Code: Bad RIP value. [312152.970430] RSP: 002b:00007f00aa6c6fc0 EFLAGS: 00010206 [312152.970432] RAX: 0000560213c4c280 RBX: 00007f00aa7c7af8 RCX: 00007f00b0b39520 [312152.970433] RDX: 0000000000000000 RSI: 0000000000000008 RDI: 00007f00aa6c6ed0 [312152.970433] RBP: 00005602204e6da0 R08: 0000000000000000 R09: 00007f00aa6c7d59 [312152.970434] R10: 00005601f82fc820 R11: 0000000000000001 R12: 00007f00aa7c7af8 [312152.970435] R13: 00005601f8bede00 R14: 0000560213c4c280 R15: 00007f00aa7c7af8 [312152.970437] memory: usage 2516992kB, limit 2516992kB, failcnt 254345 [312152.970438] memory+swap: usage 3145648kB, limit 3145728kB, failcnt 985908 [312152.970439] kmem: usage 26464kB, limit 9007199254740988kB, failcnt 0 [312152.970439] Memory cgroup stats for /docker/4570286f36c91ba31464f017556c26c70c539ee0abcc81fbaf1d7689d7615f0a: [312152.970452] anon 2550063104 file 249856 kernel_stack 1363968 slab 13627392 sock 0 shmem 0 file_mapped 270336 file_dirty 0 file_writeback 405504 anon_thp 0 inactive_anon 509853696 active_anon 2039803904 inactive_file 0 active_file 0 unevictable 0 slab_reclaimable 7442432 slab_unreclaimable 6184960 pgfault 2674155 pgmajfault 228294 workingset_refault 2706561 workingset_activate 192984 workingset_nodereclaim 0 pgrefill 2434699 pgscan 8689699 pgsteal 3128433 pgactivate 1466256 pgdeactivate 2173314 pglazyfree 0 pglazyfreed 0 thp_fault_alloc 0 thp_collapse_alloc 0 [312152.970453] Tasks state (memory values in pages): [312152.970453] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name [312152.970458] [1634573] 0 1634573 8558 3239 106496 3660 0 supervisord [312152.970460] [1635332] 1001 1635332 568756 42145 851968 24321 0 gitaly [312152.970463] [1635333] 1001 1635333 300975 48774 1892352 158917 0 ruby3.0 [312152.970464] [1635339] 1001 1635339 501544 9685 282624 571 0 gitlab-workhors [312152.970466] [1635341] 0 1635341 14691 1777 106496 801 0 nginx [312152.970468] [1635342] 0 1635342 3855 1551 65536 407 0 sshd [312152.970470] [1635356] 33 1635356 14691 448 102400 840 0 nginx [312152.970472] [1635357] 33 1635357 14691 455 102400 838 0 nginx [312152.970474] [1635358] 33 1635358 14691 424 102400 838 0 nginx [312152.970475] [1635359] 33 1635359 14691 68 102400 853 0 nginx [312152.970476] [1635360] 33 1635360 14691 220 102400 881 0 nginx [312152.970478] [1635361] 33 1635361 14691 220 102400 881 0 nginx [312152.970480] [1635549] 1001 1635549 403233 131047 2060288 92264 0 ruby3.0 [312152.970482] [1635559] 1001 1635559 403233 137637 2060288 85994 0 ruby3.0 [312152.970484] [1646271] 1001 1646271 586234 232443 2342912 0 0 ruby3.0 [312152.970486] [1646525] 1001 1646525 402405 133212 1990656 83169 0 ruby3.0 [312152.970487] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=4570286f36c91ba31464f017556c26c70c539ee0abcc81fbaf1d7689d7615f0a,mems_allowed=0,oom_memcg=/docker/4570286f36c91ba31464f017556c26c70c539ee0abcc81fbaf1d7689d7615f0a,task_memcg=/docker/4570286f36c91ba31464f017556c26c70c539ee0abcc81fbaf1d7689d7615f0a,task=ruby3.0,pid=1646271,uid=1001 [312152.970588] Memory cgroup out of memory: Killed process 1646271 (ruby3.0) total-vm:2344936kB, anon-rss:917944kB, file-rss:11828kB, shmem-rss:0kB, UID:1001 pgtables:2288kB oom_score_adj:0
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login