No, its an enterprise setup, server has 512GB of RAM, all other Cloudron apps work normally, also Wekan prior to 8.23. With version 8.22 memory usage peaks at 1GB, over a month of heavy usage.
It is not a platform issue, something was change during 8.22 to 8.23 update, either in Wekan core or Cloudron container config.
Is someone using >=8.23 version on Cloudron without issues?
@joseph I did try with 16gb - not working. Here is the log:
<--- Last few GCs --->
[19:0x280a58a0] 113560 ms: Scavenge 1172.1 (1269.6) -> 1163.6 (1269.6) MB, 128.6 / 0.3 ms (average mu = 0.990, current mu = 0.991) allocation failure
[19:0x280a58a0] 113791 ms: Scavenge 1173.1 (1269.6) -> 1166.6 (1269.6) MB, 132.6 / 0.3 ms (average mu = 0.990, current mu = 0.991) task
[19:0x280a58a0] 114066 ms: Scavenge 1180.7 (1270.9) -> 1171.7 (1270.9) MB, 134.9 / 0.4 ms (average mu = 0.990, current mu = 0.991) allocation failure
<--- JS stacktrace --->
FATAL ERROR: Zone Allocation failed - process out of memory
1: 0xa3d3a0 node::Abort() [node]
2: 0x9741f9 node::FatalError(char const*, char const*) [node]
3: 0xbbd3ee v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
4: 0xbbd767 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
5: 0x125ecee v8::internal::Zone::NewExpand(unsigned long) [node]
6: 0xfcf66a v8::internal::ExpressionScope<v8::internal::ParserTypes<v8::internal::Parser> >::NewVariable(v8::internal::AstRawString const*, int) [node]
7: 0xfde8c2 v8::internal::ParserBase<v8::internal::Parser>::ParsePrimaryExpression() [node]
8: 0xfd0bf2 v8::internal::ParserBase<v8::internal::Parser>::ParseLogicalExpression() [node]
9: 0xfd0dce v8::internal::ParserBase<v8::internal::Parser>::ParseAssignmentExpressionCoverGrammar() [node]
10: 0xfd1b7a v8::internal::ParserBase<v8::internal::Parser>::ParseArguments(v8::internal::ScopedList<v8::internal::Expression*, void*>*, bool*, v8::internal::ParsingArrowHeadFlag) [node]
11: 0xfdb872 v8::internal::ParserBase<v8::internal::Parser>::ParseLeftHandSideContinuation(v8::internal::Expression*) [node]
12: 0xfd0c30 v8::internal::ParserBase<v8::internal::Parser>::ParseLogicalExpression() [node]
13: 0xfd0dce v8::internal::ParserBase<v8::internal::Parser>::ParseAssignmentExpressionCoverGrammar() [node]
14: 0xfda9bb v8::internal::ParserBase<v8::internal::Parser>::ParseExpressionCoverGrammar() [node]
15: 0xfe4e4d v8::internal::ParserBase<v8::internal::Parser>::ParseExpressionOrLabelledStatement(v8::internal::ZoneList<v8::internal::AstRawString const*>*, v8::internal::ZoneList<v8::internal::AstRawString const*>*, v8::internal::AllowLabelledFunctionStatement) [node]
16: 0xfd412a v8::internal::ParserBase<v8::internal::Parser>::ParseStatement(v8::internal::ZoneList<v8::internal::AstRawString const*>*, v8::internal::ZoneList<v8::internal::AstRawString const*>*, v8::internal::AllowLabelledFunctionStatement) [node]
17: 0xfe1771 v8::internal::ParserBase<v8::internal::Parser>::ParseBlock(v8::internal::ZoneList<v8::internal::AstRawString const*>*) [node]
18: 0xfd413b v8::internal::ParserBase<v8::internal::Parser>::ParseStatement(v8::internal::ZoneList<v8::internal::AstRawString const*>*, v8::internal::ZoneList<v8::internal::AstRawString const*>*, v8::internal::AllowLabelledFunctionStatement) [node]
19: 0xfe5491 v8::internal::ParserBase<v8::internal::Parser>::ParseIfStatement(v8::internal::ZoneList<v8::internal::AstRawString const*>*) [node]
20: 0xfd426b v8::internal::ParserBase<v8::internal::Parser>::ParseStatement(v8::internal::ZoneList<v8::internal::AstRawString const*>*, v8::internal::ZoneList<v8::internal::AstRawString const*>*, v8::internal::AllowLabelledFunctionStatement) [node]
21: 0xfe5730 v8::internal::ParserBase<v8::internal::Parser>::ParseIfStatement(v8::internal::ZoneList<v8::internal::AstRawString const*>*) [node]
22: 0xfd426b v8::internal::ParserBase<v8::internal::Parser>::ParseStatement(v8::internal::ZoneList<v8::internal::AstRawString const*>*, v8::internal::ZoneList<v8::internal::AstRawString const*>*, v8::internal::AllowLabelledFunctionStatement) [node]
23: 0xfe1771 v8::internal::ParserBase<v8::internal::Parser>::ParseBlock(v8::internal::ZoneList<v8::internal::AstRawString const*>*) [node]
24: 0xfd413b v8::internal::ParserBase<v8::internal::Parser>::ParseStatement(v8::internal::ZoneList<v8::internal::AstRawString const*>*, v8::internal::ZoneList<v8::internal::AstRawString const*>*, v8::internal::AllowLabelledFunctionStatement) [node]
25: 0xfe5730 v8::internal::ParserBase<v8::internal::Parser>::ParseIfStatement(v8::internal::ZoneList<v8::internal::AstRawString const*>*) [node]
26: 0xfd426b v8::internal::ParserBase<v8::internal::Parser>::ParseStatement(v8::internal::ZoneList<v8::internal::AstRawString const*>*, v8::internal::ZoneList<v8::internal::AstRawString const*>*, v8::internal::AllowLabelledFunctionStatement) [node]
27: 0xfe1771 v8::internal::ParserBase<v8::internal::Parser>::ParseBlock(v8::internal::ZoneList<v8::internal::AstRawString const*>*) [node]
28: 0xfd413b v8::internal::ParserBase<v8::internal::Parser>::ParseStatement(v8::internal::ZoneList<v8::internal::AstRawString const*>*, v8::internal::ZoneList<v8::internal::AstRawString const*>*, v8::internal::AllowLabelledFunctionStatement) [node]
29: 0xfe5491 v8::internal::ParserBase<v8::internal::Parser>::ParseIfStatement(v8::internal::ZoneList<v8::internal::AstRawString const*>*) [node]
30: 0xfd426b v8::internal::ParserBase<v8::internal::Parser>::ParseStatement(v8::internal::ZoneList<v8::internal::AstRawString const*>*, v8::internal::ZoneList<v8::internal::AstRawString const*>*, v8::internal::AllowLabelledFunctionStatement) [node]
31: 0xfd66c0 v8::internal::ParserBase<v8::internal::Parser>::ParseFunctionBody(v8::internal::ScopedList<v8::internal::Statement*, void*>*, v8::internal::AstRawString const*, int, v8::internal::ParserFormalParameters const&, v8::internal::FunctionKind, v8::internal::FunctionSyntaxKind, v8::internal::ParserBase<v8::internal::Parser>::FunctionBodyType) [node]
32: 0xfd7ce0 v8::internal::Parser::ParseFunction(v8::internal::ScopedList<v8::internal::Statement*, void*>*, v8::internal::AstRawString const*, int, v8::internal::FunctionKind, v8::internal::FunctionSyntaxKind, v8::internal::DeclarationScope*, int*, int*, bool*, int*, int*, v8::internal::ZoneList<v8::internal::AstRawString const*>*) [node]
33: 0xfd82cd v8::internal::Parser::ParseFunctionLiteral(v8::internal::AstRawString const*, v8::internal::Scanner::Location, v8::internal::FunctionNameValidity, v8::internal::FunctionKind, int, v8::internal::FunctionSyntaxKind, v8::internal::LanguageMode, v8::internal::ZoneList<v8::internal::AstRawString const*>*) [node]
34: 0xfe02f7 v8::internal::Parser::DoParseFunction(v8::internal::Isolate*, v8::internal::ParseInfo*, int, int, int, v8::internal::AstRawString const*) [node]
35: 0xfe0eab v8::internal::Parser::ParseFunction(v8::internal::Isolate*, v8::internal::ParseInfo*, v8::internal::Handle<v8::internal::SharedFunctionInfo>) [node]
36: 0xfe5c81 v8::internal::parsing::ParseFunction(v8::internal::ParseInfo*, v8::internal::Handle<v8::internal::SharedFunctionInfo>, v8::internal::Isolate*, v8::internal::parsing::ReportErrorsAndStatisticsMode) [node]
37: 0xfe60a5 v8::internal::parsing::ParseAny(v8::internal::ParseInfo*, v8::internal::Handle<v8::internal::SharedFunctionInfo>, v8::internal::Isolate*, v8::internal::parsing::ReportErrorsAndStatisticsMode) [node]
38: 0xc8a800 v8::internal::Compiler::Compile(v8::internal::Handle<v8::internal::SharedFunctionInfo>, v8::internal::Compiler::ClearExceptionFlag, v8::internal::IsCompiledScope*) [node]
39: 0xc8c99c v8::internal::Compiler::Compile(v8::internal::Handle<v8::internal::JSFunction>, v8::internal::Compiler::ClearExceptionFlag, v8::internal::IsCompiledScope*) [node]
40: 0x1085b7a v8::internal::Runtime_CompileLazy(int, unsigned long*, v8::internal::Isolate*) [node]
41: 0x1440959 [node]
Aborted (core dumped)
@humptydumpty indeed, I was a surprised by the attitude - blaming kernel, hardware and similar without any research into matter when they are already fully aware how many issues are happening with the new major version.
@joseph Thank you, I will try that. Is it possible to increase node heap limit?
I am writing here as wekan community has been very uninterested to deal with this - https://github.com/wekan/wekan/issues/6132
Versions tested:
8.22: stable (boards open normally)
8.28: boards never open (infinite loading), server restarts/crashes
8.31: issue still present
Platform: Cloudron (Docker-based), Ubuntu host
Symptoms:
Web UI loads, but opening any board spins forever.
Backend restarts repeatedly; healthcheck fails (ECONNREFUSED :3000) after crash.
Logs:
terminate called after throwing an instance of 'std::bad_alloc' what(): std::bad_alloc
also observed V8 OOM: FATAL ERROR: Zone Allocation failed - process out of memory
Container resources:
/sys/fs/cgroup/memory.max = 17179869184 (16GB)
/sys/fs/cgroup/memory.swap.max = max
Node heap limit (default) shows ~4144MB via require('v8').getHeapStatistics().heap_size_limit
Notes / suspicion:
Works on 8.22; regression appears after 8.23/8.24 era.
v8.23 changed publish layer (cottz:publish-relations → reywood:publish-composite).
v8.24 included Meteor 2.16 updates, async index creation, removed idmontie:migrations, etc.
v8.28 updated meteor-node-stubs.
There is an existing report of std::bad_alloc restart loop after upgrading 8.23 → 8.24 (#6098).
Happy to run additional diagnostics if you suggest flags or debug logging. I tried increasing allocated memory to 64GB or even higher, but result is the same.
Hi,
--apply-db-migrations had the same error before updating tasks table. I inserted column because of this repeating error in box.log:
2025-12-07T15:23:51.233Z box:server ========================================== 2025-12-07T15:23:51.234Z box:server Cloudron 9.0.13
2025-12-07T15:23:51.234Z box:server ========================================== 2025-12-07T15:23:51.234Z box:platform initialize: start platform 2025-12-07T15:23:51.236Z box:tasks stopAllTasks: 0 tasks are running. sending abort signal 2025-12-07T15:23:51.236Z box:shell tasks: /usr/bin/sudo --non-interactive /home/yellowtent/box/src/scripts/stoptask.sh all 2025-12-07T15:23:51.293Z box:locks releaseAll: all locks released 2025-12-07T15:23:51.297Z box:reverseproxy writeDashboardConfig: writing dashboard config for cloud.voxellab.rs 2025-12-07T15:23:51.309Z box:shell reverseproxy: /usr/bin/sudo --non-interactive /home/yellowtent/box/src/scripts/restartservice.sh nginx 2025-12-07T15:23:51.646Z box:updater notifyBoxUpdate: update finished from 8.3.2 to 9.0.13 2025-12-07T15:23:51.648Z Error starting servers 2025-12-07T15:23:51.648Z BoxError: Unknown column 'completed' in 'field list' 2025-12-07T15:23:51.648Z at Object.query (/home/yellowtent/box/src/database.js:96:22) 2025-12-07T15:23:51.648Z at process.processTicksAndRejections (node:internal/process/task_queues:105:5) 2025-12-07T15:23:51.648Z at async list (/home/yellowtent/box/src/tasks.js:163:21) 2025-12-07T15:23:51.648Z at async Object.setCompletedByType (/home/yellowtent/box/src/tasks.js:172:21)
I referred to forum post here where user had missing pending column after the same version upgrade.
Anyway, I used this:
USE box;
SHOW COLUMNS FROM tasks;
ALTER TABLE tasks
ADD COLUMN completed TINYINT(1) NOT NULL DEFAULT 0;
SHOW COLUMNS FROM tasks;
This restored working dashboard and my server is operational at the moment, but I should resolve the oidc issue at some point.
Thanks
After upgrade from 8 to 9.0.13 dashboard not working. With the hep of other post, I noticed my tasks table doesn't have completed column, so adding it manually solved the issue with dashboard not loading.
After running troubleshoot script:
[FAIL] Database migrations are pending. Last migration in DB: /20250618200400-fix-charset-collate.js. Last migration file: /20251110084515-cloudflare-add-defaultProxyStatus.js.
Please run 'cloudron-support --apply-db-migrations' to apply the migrations.
After running --apply-db-migrations:
[ERROR] AssertionError [ERR_ASSERTION]: ifError got unwanted exception: Duplicate entry '20647655-d757-412a-8a67-3b6ff3d224a3-oidc' for key 'oidcClients.PRIMARY'
at /home/yellowtent/box/node_modules/db-migrate/lib/commands/on-complete.js:15:14
at tryCatcher (/home/yellowtent/box/node_modules/bluebird/js/release/util.js:16:23)
at Promise.successAdapter (/home/yellowtent/box/node_modules/bluebird/js/release/nodeify.js:22:30)
at Promise._settlePromise (/home/yellowtent/box/node_modules/bluebird/js/release/promise.js:601:21)
at Promise._settlePromiseCtx (/home/yellowtent/box/node_modules/bluebird/js/release/promise.js:641:10)
at _drainQueueStep (/home/yellowtent/box/node_modules/bluebird/js/release/async.js:97:12)
at _drainQueue (/home/yellowtent/box/node_modules/bluebird/js/release/async.js:86:9)
at Async._drainQueues (/home/yellowtent/box/node_modules/bluebird/js/release/async.js:102:5)
at Async.drainQueues [as _onImmediate] (/home/yellowtent/box/node_modules/bluebird/js/release/async.js:15:14)
at process.processImmediate (node:internal/timers:485:21)
at Packet.asError (/home/yellowtent/box/node_modules/mysql2/lib/packets/packet.js:740:17)
at Query.execute (/home/yellowtent/box/node_modules/mysql2/lib/commands/command.js:29:26)
at Connection.handlePacket (/home/yellowtent/box/node_modules/mysql2/lib/base/connection.js:477:34)
at PacketParser.onPacket (/home/yellowtent/box/node_modules/mysql2/lib/base/connection.js:93:12)
at PacketParser.executeStart (/home/yellowtent/box/node_modules/mysql2/lib/packet_parser.js:75:16)
at Socket.<anonymous> (/home/yellowtent/box/node_modules/mysql2/lib/base/connection.js:100:25)
at Socket.emit (node:events:519:28)
at addChunk (node:internal/streams/readable:561:12)
at readableAddChunkPushByteMode (node:internal/streams/readable:512:3)
at Readable.push (node:internal/streams/readable:392:5)
at TCP.onStreamRead (node:internal/stream_base_commons:189:23)
2025-12-07T16:47:15 ==> start: DB migration failed
[FAIL] Database migrations failed
9.0.13