Boards never open; Wekan restarts with std::bad_alloc / V8 OOM on large datasets (works on 8.22)

[dd4y]

I am writing here as wekan community has been very uninterested to deal with this - https://github.com/wekan/wekan/issues/6132

Versions tested:

8.22: stable (boards open normally)

8.28: boards never open (infinite loading), server restarts/crashes

8.31: issue still present

Platform: Cloudron (Docker-based), Ubuntu host

Symptoms:

Web UI loads, but opening any board spins forever.

Backend restarts repeatedly; healthcheck fails (ECONNREFUSED :3000) after crash.

Logs:

terminate called after throwing an instance of 'std::bad_alloc' what(): std::bad_alloc

also observed V8 OOM: FATAL ERROR: Zone Allocation failed - process out of memory

Container resources:

/sys/fs/cgroup/memory.max = 17179869184 (16GB)

/sys/fs/cgroup/memory.swap.max = max

Node heap limit (default) shows ~4144MB via require('v8').getHeapStatistics().heap_size_limit

Notes / suspicion:

Works on 8.22; regression appears after 8.23/8.24 era.

v8.23 changed publish layer (cottz:publish-relations → reywood:publish-composite).

v8.24 included Meteor 2.16 updates, async index creation, removed idmontie:migrations, etc.

v8.28 updated meteor-node-stubs.

There is an existing report of std::bad_alloc restart loop after upgrading 8.23 → 8.24 (#6098).

Happy to run additional diagnostics if you suggest flags or debug logging. I tried increasing allocated memory to 64GB or even higher, but result is the same.

[humpty]

I can’t help with your troubleshooting, but xet7’s handling of the latest UI changes drove me away. If you’re not using rules and automations, give Vikunja a try.

@scooke ‘s very cool use case if you’d like to give it a read https://forum.cloudron.io/topic/5693/domain-and-vps-records-management-using-vikunja-lengthy-read

[joseph]

64GB seems like a ridiculous amount of space for a web app!

@dd4y Can you this:

• Put the app in recovery mode
• Open web terminal
• export NODE_OPTIONS="--max-old-space-size=4096"
• Now run , /app/pkg/start.sh

I think that value can go up to 16GB . Can you try by increasing that value till it works ?

[dd4y]

@humptydumpty indeed, I was a surprised by the attitude - blaming kernel, hardware and similar without any research into matter when they are already fully aware how many issues are happening with the new major version.

@joseph Thank you, I will try that. Is it possible to increase node heap limit?

[joseph]

@dd4y if the instructions I gave in the previous post worked, we can adjust the package to use a larger heap size by default (heap size is adjusted using max-old-space)

[dd4y]

@joseph I did try with 16gb - not working. Here is the log:

<--- Last few GCs --->

[19:0x280a58a0]   113560 ms: Scavenge 1172.1 (1269.6) -> 1163.6 (1269.6) MB, 128.6 / 0.3 ms  (average mu = 0.990, current mu = 0.991) allocation failure 
[19:0x280a58a0]   113791 ms: Scavenge 1173.1 (1269.6) -> 1166.6 (1269.6) MB, 132.6 / 0.3 ms  (average mu = 0.990, current mu = 0.991) task 
[19:0x280a58a0]   114066 ms: Scavenge 1180.7 (1270.9) -> 1171.7 (1270.9) MB, 134.9 / 0.4 ms  (average mu = 0.990, current mu = 0.991) allocation failure 


<--- JS stacktrace --->

FATAL ERROR: Zone Allocation failed - process out of memory
 1: 0xa3d3a0 node::Abort() [node]
 2: 0x9741f9 node::FatalError(char const*, char const*) [node]
 3: 0xbbd3ee v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
 4: 0xbbd767 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
 5: 0x125ecee v8::internal::Zone::NewExpand(unsigned long) [node]
 6: 0xfcf66a v8::internal::ExpressionScope<v8::internal::ParserTypes<v8::internal::Parser> >::NewVariable(v8::internal::AstRawString const*, int) [node]
 7: 0xfde8c2 v8::internal::ParserBase<v8::internal::Parser>::ParsePrimaryExpression() [node]
 8: 0xfd0bf2 v8::internal::ParserBase<v8::internal::Parser>::ParseLogicalExpression() [node]
 9: 0xfd0dce v8::internal::ParserBase<v8::internal::Parser>::ParseAssignmentExpressionCoverGrammar() [node]
10: 0xfd1b7a v8::internal::ParserBase<v8::internal::Parser>::ParseArguments(v8::internal::ScopedList<v8::internal::Expression*, void*>*, bool*, v8::internal::ParsingArrowHeadFlag) [node]
11: 0xfdb872 v8::internal::ParserBase<v8::internal::Parser>::ParseLeftHandSideContinuation(v8::internal::Expression*) [node]
12: 0xfd0c30 v8::internal::ParserBase<v8::internal::Parser>::ParseLogicalExpression() [node]
13: 0xfd0dce v8::internal::ParserBase<v8::internal::Parser>::ParseAssignmentExpressionCoverGrammar() [node]
14: 0xfda9bb v8::internal::ParserBase<v8::internal::Parser>::ParseExpressionCoverGrammar() [node]
15: 0xfe4e4d v8::internal::ParserBase<v8::internal::Parser>::ParseExpressionOrLabelledStatement(v8::internal::ZoneList<v8::internal::AstRawString const*>*, v8::internal::ZoneList<v8::internal::AstRawString const*>*, v8::internal::AllowLabelledFunctionStatement) [node]
16: 0xfd412a v8::internal::ParserBase<v8::internal::Parser>::ParseStatement(v8::internal::ZoneList<v8::internal::AstRawString const*>*, v8::internal::ZoneList<v8::internal::AstRawString const*>*, v8::internal::AllowLabelledFunctionStatement) [node]
17: 0xfe1771 v8::internal::ParserBase<v8::internal::Parser>::ParseBlock(v8::internal::ZoneList<v8::internal::AstRawString const*>*) [node]
18: 0xfd413b v8::internal::ParserBase<v8::internal::Parser>::ParseStatement(v8::internal::ZoneList<v8::internal::AstRawString const*>*, v8::internal::ZoneList<v8::internal::AstRawString const*>*, v8::internal::AllowLabelledFunctionStatement) [node]
19: 0xfe5491 v8::internal::ParserBase<v8::internal::Parser>::ParseIfStatement(v8::internal::ZoneList<v8::internal::AstRawString const*>*) [node]
20: 0xfd426b v8::internal::ParserBase<v8::internal::Parser>::ParseStatement(v8::internal::ZoneList<v8::internal::AstRawString const*>*, v8::internal::ZoneList<v8::internal::AstRawString const*>*, v8::internal::AllowLabelledFunctionStatement) [node]
21: 0xfe5730 v8::internal::ParserBase<v8::internal::Parser>::ParseIfStatement(v8::internal::ZoneList<v8::internal::AstRawString const*>*) [node]
22: 0xfd426b v8::internal::ParserBase<v8::internal::Parser>::ParseStatement(v8::internal::ZoneList<v8::internal::AstRawString const*>*, v8::internal::ZoneList<v8::internal::AstRawString const*>*, v8::internal::AllowLabelledFunctionStatement) [node]
23: 0xfe1771 v8::internal::ParserBase<v8::internal::Parser>::ParseBlock(v8::internal::ZoneList<v8::internal::AstRawString const*>*) [node]
24: 0xfd413b v8::internal::ParserBase<v8::internal::Parser>::ParseStatement(v8::internal::ZoneList<v8::internal::AstRawString const*>*, v8::internal::ZoneList<v8::internal::AstRawString const*>*, v8::internal::AllowLabelledFunctionStatement) [node]
25: 0xfe5730 v8::internal::ParserBase<v8::internal::Parser>::ParseIfStatement(v8::internal::ZoneList<v8::internal::AstRawString const*>*) [node]
26: 0xfd426b v8::internal::ParserBase<v8::internal::Parser>::ParseStatement(v8::internal::ZoneList<v8::internal::AstRawString const*>*, v8::internal::ZoneList<v8::internal::AstRawString const*>*, v8::internal::AllowLabelledFunctionStatement) [node]
27: 0xfe1771 v8::internal::ParserBase<v8::internal::Parser>::ParseBlock(v8::internal::ZoneList<v8::internal::AstRawString const*>*) [node]
28: 0xfd413b v8::internal::ParserBase<v8::internal::Parser>::ParseStatement(v8::internal::ZoneList<v8::internal::AstRawString const*>*, v8::internal::ZoneList<v8::internal::AstRawString const*>*, v8::internal::AllowLabelledFunctionStatement) [node]
29: 0xfe5491 v8::internal::ParserBase<v8::internal::Parser>::ParseIfStatement(v8::internal::ZoneList<v8::internal::AstRawString const*>*) [node]
30: 0xfd426b v8::internal::ParserBase<v8::internal::Parser>::ParseStatement(v8::internal::ZoneList<v8::internal::AstRawString const*>*, v8::internal::ZoneList<v8::internal::AstRawString const*>*, v8::internal::AllowLabelledFunctionStatement) [node]
31: 0xfd66c0 v8::internal::ParserBase<v8::internal::Parser>::ParseFunctionBody(v8::internal::ScopedList<v8::internal::Statement*, void*>*, v8::internal::AstRawString const*, int, v8::internal::ParserFormalParameters const&, v8::internal::FunctionKind, v8::internal::FunctionSyntaxKind, v8::internal::ParserBase<v8::internal::Parser>::FunctionBodyType) [node]
32: 0xfd7ce0 v8::internal::Parser::ParseFunction(v8::internal::ScopedList<v8::internal::Statement*, void*>*, v8::internal::AstRawString const*, int, v8::internal::FunctionKind, v8::internal::FunctionSyntaxKind, v8::internal::DeclarationScope*, int*, int*, bool*, int*, int*, v8::internal::ZoneList<v8::internal::AstRawString const*>*) [node]
33: 0xfd82cd v8::internal::Parser::ParseFunctionLiteral(v8::internal::AstRawString const*, v8::internal::Scanner::Location, v8::internal::FunctionNameValidity, v8::internal::FunctionKind, int, v8::internal::FunctionSyntaxKind, v8::internal::LanguageMode, v8::internal::ZoneList<v8::internal::AstRawString const*>*) [node]
34: 0xfe02f7 v8::internal::Parser::DoParseFunction(v8::internal::Isolate*, v8::internal::ParseInfo*, int, int, int, v8::internal::AstRawString const*) [node]
35: 0xfe0eab v8::internal::Parser::ParseFunction(v8::internal::Isolate*, v8::internal::ParseInfo*, v8::internal::Handle<v8::internal::SharedFunctionInfo>) [node]
36: 0xfe5c81 v8::internal::parsing::ParseFunction(v8::internal::ParseInfo*, v8::internal::Handle<v8::internal::SharedFunctionInfo>, v8::internal::Isolate*, v8::internal::parsing::ReportErrorsAndStatisticsMode) [node]
37: 0xfe60a5 v8::internal::parsing::ParseAny(v8::internal::ParseInfo*, v8::internal::Handle<v8::internal::SharedFunctionInfo>, v8::internal::Isolate*, v8::internal::parsing::ReportErrorsAndStatisticsMode) [node]
38: 0xc8a800 v8::internal::Compiler::Compile(v8::internal::Handle<v8::internal::SharedFunctionInfo>, v8::internal::Compiler::ClearExceptionFlag, v8::internal::IsCompiledScope*) [node]
39: 0xc8c99c v8::internal::Compiler::Compile(v8::internal::Handle<v8::internal::JSFunction>, v8::internal::Compiler::ClearExceptionFlag, v8::internal::IsCompiledScope*) [node]
40: 0x1085b7a v8::internal::Runtime_CompileLazy(int, unsigned long*, v8::internal::Isolate*) [node]
41: 0x1440959  [node]
Aborted (core dumped)

[nebulon]

How much memory does the whole server have? Maybe it overall runs out of memory, so non of the limits set are even hit?

[dd4y]

No, its an enterprise setup, server has 512GB of RAM, all other Cloudron apps work normally, also Wekan prior to 8.23. With version 8.22 memory usage peaks at 1GB, over a month of heavy usage.

It is not a platform issue, something was change during 8.22 to 8.23 update, either in Wekan core or Cloudron container config.

Is someone using >=8.23 version on Cloudron without issues?

[nebulon]

Unless you did some Cloudron update between the working and broken wekan, nothing would have changed on Cloudron side's container config as such.

I guess this would require some bisecting of the changes in wekan between those releases. Not all too much has changed according to https://github.com/wekan/wekan/releases/tag/v8.23 but we are not really wekan experts either.