Boards never open; Wekan restarts with std::bad_alloc / V8 OOM on large datasets (works on 8.22)

[joseph]

64GB seems like a ridiculous amount of space for a web app!

@dd4y Can you this:

• Put the app in recovery mode
• Open web terminal
• export NODE_OPTIONS="--max-old-space-size=4096"
• Now run , /app/pkg/start.sh

I think that value can go up to 16GB . Can you try by increasing that value till it works ?

[dd4y]

@humptydumpty indeed, I was a surprised by the attitude - blaming kernel, hardware and similar without any research into matter when they are already fully aware how many issues are happening with the new major version.

@joseph Thank you, I will try that. Is it possible to increase node heap limit?

[joseph]

@dd4y if the instructions I gave in the previous post worked, we can adjust the package to use a larger heap size by default (heap size is adjusted using max-old-space)

[dd4y]

@joseph I did try with 16gb - not working. Here is the log:

<--- Last few GCs --->

[19:0x280a58a0]   113560 ms: Scavenge 1172.1 (1269.6) -> 1163.6 (1269.6) MB, 128.6 / 0.3 ms  (average mu = 0.990, current mu = 0.991) allocation failure 
[19:0x280a58a0]   113791 ms: Scavenge 1173.1 (1269.6) -> 1166.6 (1269.6) MB, 132.6 / 0.3 ms  (average mu = 0.990, current mu = 0.991) task 
[19:0x280a58a0]   114066 ms: Scavenge 1180.7 (1270.9) -> 1171.7 (1270.9) MB, 134.9 / 0.4 ms  (average mu = 0.990, current mu = 0.991) allocation failure 


<--- JS stacktrace --->

FATAL ERROR: Zone Allocation failed - process out of memory
 1: 0xa3d3a0 node::Abort() [node]
 2: 0x9741f9 node::FatalError(char const*, char const*) [node]
 3: 0xbbd3ee v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
 4: 0xbbd767 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
 5: 0x125ecee v8::internal::Zone::NewExpand(unsigned long) [node]
 6: 0xfcf66a v8::internal::ExpressionScope<v8::internal::ParserTypes<v8::internal::Parser> >::NewVariable(v8::internal::AstRawString const*, int) [node]
 7: 0xfde8c2 v8::internal::ParserBase<v8::internal::Parser>::ParsePrimaryExpression() [node]
 8: 0xfd0bf2 v8::internal::ParserBase<v8::internal::Parser>::ParseLogicalExpression() [node]
 9: 0xfd0dce v8::internal::ParserBase<v8::internal::Parser>::ParseAssignmentExpressionCoverGrammar() [node]
10: 0xfd1b7a v8::internal::ParserBase<v8::internal::Parser>::ParseArguments(v8::internal::ScopedList<v8::internal::Expression*, void*>*, bool*, v8::internal::ParsingArrowHeadFlag) [node]
11: 0xfdb872 v8::internal::ParserBase<v8::internal::Parser>::ParseLeftHandSideContinuation(v8::internal::Expression*) [node]
12: 0xfd0c30 v8::internal::ParserBase<v8::internal::Parser>::ParseLogicalExpression() [node]
13: 0xfd0dce v8::internal::ParserBase<v8::internal::Parser>::ParseAssignmentExpressionCoverGrammar() [node]
14: 0xfda9bb v8::internal::ParserBase<v8::internal::Parser>::ParseExpressionCoverGrammar() [node]
15: 0xfe4e4d v8::internal::ParserBase<v8::internal::Parser>::ParseExpressionOrLabelledStatement(v8::internal::ZoneList<v8::internal::AstRawString const*>*, v8::internal::ZoneList<v8::internal::AstRawString const*>*, v8::internal::AllowLabelledFunctionStatement) [node]
16: 0xfd412a v8::internal::ParserBase<v8::internal::Parser>::ParseStatement(v8::internal::ZoneList<v8::internal::AstRawString const*>*, v8::internal::ZoneList<v8::internal::AstRawString const*>*, v8::internal::AllowLabelledFunctionStatement) [node]
17: 0xfe1771 v8::internal::ParserBase<v8::internal::Parser>::ParseBlock(v8::internal::ZoneList<v8::internal::AstRawString const*>*) [node]
18: 0xfd413b v8::internal::ParserBase<v8::internal::Parser>::ParseStatement(v8::internal::ZoneList<v8::internal::AstRawString const*>*, v8::internal::ZoneList<v8::internal::AstRawString const*>*, v8::internal::AllowLabelledFunctionStatement) [node]
19: 0xfe5491 v8::internal::ParserBase<v8::internal::Parser>::ParseIfStatement(v8::internal::ZoneList<v8::internal::AstRawString const*>*) [node]
20: 0xfd426b v8::internal::ParserBase<v8::internal::Parser>::ParseStatement(v8::internal::ZoneList<v8::internal::AstRawString const*>*, v8::internal::ZoneList<v8::internal::AstRawString const*>*, v8::internal::AllowLabelledFunctionStatement) [node]
21: 0xfe5730 v8::internal::ParserBase<v8::internal::Parser>::ParseIfStatement(v8::internal::ZoneList<v8::internal::AstRawString const*>*) [node]
22: 0xfd426b v8::internal::ParserBase<v8::internal::Parser>::ParseStatement(v8::internal::ZoneList<v8::internal::AstRawString const*>*, v8::internal::ZoneList<v8::internal::AstRawString const*>*, v8::internal::AllowLabelledFunctionStatement) [node]
23: 0xfe1771 v8::internal::ParserBase<v8::internal::Parser>::ParseBlock(v8::internal::ZoneList<v8::internal::AstRawString const*>*) [node]
24: 0xfd413b v8::internal::ParserBase<v8::internal::Parser>::ParseStatement(v8::internal::ZoneList<v8::internal::AstRawString const*>*, v8::internal::ZoneList<v8::internal::AstRawString const*>*, v8::internal::AllowLabelledFunctionStatement) [node]
25: 0xfe5730 v8::internal::ParserBase<v8::internal::Parser>::ParseIfStatement(v8::internal::ZoneList<v8::internal::AstRawString const*>*) [node]
26: 0xfd426b v8::internal::ParserBase<v8::internal::Parser>::ParseStatement(v8::internal::ZoneList<v8::internal::AstRawString const*>*, v8::internal::ZoneList<v8::internal::AstRawString const*>*, v8::internal::AllowLabelledFunctionStatement) [node]
27: 0xfe1771 v8::internal::ParserBase<v8::internal::Parser>::ParseBlock(v8::internal::ZoneList<v8::internal::AstRawString const*>*) [node]
28: 0xfd413b v8::internal::ParserBase<v8::internal::Parser>::ParseStatement(v8::internal::ZoneList<v8::internal::AstRawString const*>*, v8::internal::ZoneList<v8::internal::AstRawString const*>*, v8::internal::AllowLabelledFunctionStatement) [node]
29: 0xfe5491 v8::internal::ParserBase<v8::internal::Parser>::ParseIfStatement(v8::internal::ZoneList<v8::internal::AstRawString const*>*) [node]
30: 0xfd426b v8::internal::ParserBase<v8::internal::Parser>::ParseStatement(v8::internal::ZoneList<v8::internal::AstRawString const*>*, v8::internal::ZoneList<v8::internal::AstRawString const*>*, v8::internal::AllowLabelledFunctionStatement) [node]
31: 0xfd66c0 v8::internal::ParserBase<v8::internal::Parser>::ParseFunctionBody(v8::internal::ScopedList<v8::internal::Statement*, void*>*, v8::internal::AstRawString const*, int, v8::internal::ParserFormalParameters const&, v8::internal::FunctionKind, v8::internal::FunctionSyntaxKind, v8::internal::ParserBase<v8::internal::Parser>::FunctionBodyType) [node]
32: 0xfd7ce0 v8::internal::Parser::ParseFunction(v8::internal::ScopedList<v8::internal::Statement*, void*>*, v8::internal::AstRawString const*, int, v8::internal::FunctionKind, v8::internal::FunctionSyntaxKind, v8::internal::DeclarationScope*, int*, int*, bool*, int*, int*, v8::internal::ZoneList<v8::internal::AstRawString const*>*) [node]
33: 0xfd82cd v8::internal::Parser::ParseFunctionLiteral(v8::internal::AstRawString const*, v8::internal::Scanner::Location, v8::internal::FunctionNameValidity, v8::internal::FunctionKind, int, v8::internal::FunctionSyntaxKind, v8::internal::LanguageMode, v8::internal::ZoneList<v8::internal::AstRawString const*>*) [node]
34: 0xfe02f7 v8::internal::Parser::DoParseFunction(v8::internal::Isolate*, v8::internal::ParseInfo*, int, int, int, v8::internal::AstRawString const*) [node]
35: 0xfe0eab v8::internal::Parser::ParseFunction(v8::internal::Isolate*, v8::internal::ParseInfo*, v8::internal::Handle<v8::internal::SharedFunctionInfo>) [node]
36: 0xfe5c81 v8::internal::parsing::ParseFunction(v8::internal::ParseInfo*, v8::internal::Handle<v8::internal::SharedFunctionInfo>, v8::internal::Isolate*, v8::internal::parsing::ReportErrorsAndStatisticsMode) [node]
37: 0xfe60a5 v8::internal::parsing::ParseAny(v8::internal::ParseInfo*, v8::internal::Handle<v8::internal::SharedFunctionInfo>, v8::internal::Isolate*, v8::internal::parsing::ReportErrorsAndStatisticsMode) [node]
38: 0xc8a800 v8::internal::Compiler::Compile(v8::internal::Handle<v8::internal::SharedFunctionInfo>, v8::internal::Compiler::ClearExceptionFlag, v8::internal::IsCompiledScope*) [node]
39: 0xc8c99c v8::internal::Compiler::Compile(v8::internal::Handle<v8::internal::JSFunction>, v8::internal::Compiler::ClearExceptionFlag, v8::internal::IsCompiledScope*) [node]
40: 0x1085b7a v8::internal::Runtime_CompileLazy(int, unsigned long*, v8::internal::Isolate*) [node]
41: 0x1440959  [node]
Aborted (core dumped)

[nebulon]

How much memory does the whole server have? Maybe it overall runs out of memory, so non of the limits set are even hit?

[dd4y]

No, its an enterprise setup, server has 512GB of RAM, all other Cloudron apps work normally, also Wekan prior to 8.23. With version 8.22 memory usage peaks at 1GB, over a month of heavy usage.

It is not a platform issue, something was change during 8.22 to 8.23 update, either in Wekan core or Cloudron container config.

Is someone using >=8.23 version on Cloudron without issues?

[nebulon]

Unless you did some Cloudron update between the working and broken wekan, nothing would have changed on Cloudron side's container config as such.

I guess this would require some bisecting of the changes in wekan between those releases. Not all too much has changed according to https://github.com/wekan/wekan/releases/tag/v8.23 but we are not really wekan experts either.

[dd4y]

@nebulon Can you please update current Wekan package? This version is not stable and has issues. Btw problem I had earlier was resolved by newer versions. so it was a wekan issue after all.

[nebulon]

We will see if we can get the newer versions to work. We had stopped delivering wekan updates as upstream was blazing away with too many unstable changes and we have little wekan usage amongst Cloudron users. Lets see if we can get the latest ones to run more reliably.

[nebulon]

We managed to get the package refactored to work again with latest Wekan releases. So the latest version got now released