Going to "system info" triggers assertion error

[fbartels]

Hi,

I've already noticed this a while ago, but never bothered to look into it until now.

When going to the "system info" as an admin I get a brief toast notification that my Cloudron would be offline (triggered by a 502 return code on both /api/v1/cloudron/disks and /api/v1/cloudron/status.

Looking into the box logs the following is logged at the time of the error message:

Mar 15 16:56:30 box:apphealthmonitor app health: 24 alive / 0 dead.
AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:
actual - expected
'undefined'
'string'
at getAppDisks (/home/yellowtent/box/src/system.js:48:12)
at /home/yellowtent/box/node_modules/async/dist/async.js:2948:28
at replenish (/home/yellowtent/box/node_modules/async/dist/async.js:440:21)
at /home/yellowtent/box/node_modules/async/dist/async.js:445:13
at eachOfLimit$1 (/home/yellowtent/box/node_modules/async/dist/async.js:471:34)
at awaitable (/home/yellowtent/box/node_modules/async/dist/async.js:208:32)
at eachOfSeries (/home/yellowtent/box/node_modules/async/dist/async.js:658:16)
at awaitable (/home/yellowtent/box/node_modules/async/dist/async.js:208:32)
at /home/yellowtent/box/node_modules/async/dist/async.js:2947:9
at awaitable (/home/yellowtent/box/node_modules/async/dist/async.js:208:32) [ /home/yellowtent/box/box.js:61:17 ]
Mar 15 16:56:42 box:server ==========================================
Mar 15 16:56:42 box:server Cloudron 6.2.4
Mar 15 16:56:42 box:server ==========================================
Mar 15 16:56:42 box:settings initCache: pre-load settings
Mar 15 16:56:42 box:tasks stopTask: stopping all tasks

My Cloudron still runs on an Ubuntu 16.04 machine, maybe an incompatibility with the Node version on that system?

[nebulon]

@fbartels hm tracing the code there, this seems to be the result of getting the diskfree stats for the apps data dir, which is /home/yellowtent/appsdata. Is that by any chance a symlink or otherwise different, like a mountpoint?

[fbartels]

Hi @nebulon,

no I cannot really see something out of the ordinary with that folder. All data is on a single partition on that server.

root@my:/home/yellowtent# ls -la
insgesamt 48
drwxr-xr-x  8 yellowtent yellowtent 4096 Mär 13 02:27 .
drwxr-xr-x  4 root       root       4096 Feb  6  2018 ..
drwxr-xr-x 26 yellowtent yellowtent 4096 Mär  9 12:32 appsdata
-rw-r--r--  1 yellowtent yellowtent  220 Aug 31  2015 .bash_logout
-rw-r--r--  1 yellowtent yellowtent 3771 Aug 31  2015 .bashrc
drwxr-xr-x 10 yellowtent yellowtent 4096 Mär 12 03:01 box
drwxr-xr-x 10 yellowtent yellowtent 4096 Dez  7 02:25 boxdata
drwxr-xr-x  2 yellowtent yellowtent 4096 Dez 14  2018 configs
drwx------  2 yellowtent yellowtent 4096 Aug 19  2018 .gnupg
drwxr-xr-x 16 yellowtent yellowtent 4096 Mär  9 12:28 platformdata
-rw-r--r--  1 yellowtent yellowtent  655 Mai 16  2017 .profile
-rw-------  1 yellowtent yellowtent 1024 Feb 23 13:32 .rnd

[girish]

@fbartels Can you try this node repl session? You have to cd /home/yellowtent/box first.

root@my:/home/yellowtent/box# node 
Welcome to Node.js v14.15.4.
Type ".help" for more information.
> df = require('@sindresorhus/df')
<ref *1> [AsyncFunction: df] {
  fs: [AsyncFunction (anonymous)],
  file: [AsyncFunction (anonymous)],
  default: [Circular *1]
}
> df.file('/home/yellowtent/appsdata').then(x => console.log(x))
Promise { <pending> }
> {
  filesystem: '/dev/vda1',
  type: 'ext4',
  size: 51848359936,
  used: 12759597056,
  available: 39071985664,
  capacity: 0.25,
  mountpoint: '/'
}

For some reason, I think on your server it's returning undefined.

[fbartels]

Hi @girish,

its indeed returning "undefined"

df = require('@sindresorhus/df')
<ref *1> [AsyncFunction: df] {
  fs: [AsyncFunction (anonymous)],
  file: [AsyncFunction (anonymous)],
  default: [Circular *1]
}
> df.file('/home/yellowtent/appsdata').then(x => console.log(x))
Promise { <pending> }
> {
  filesystem: undefined,
  type: undefined,
  size: NaN,
  used: NaN,
  available: NaN,
  capacity: NaN,
  mountpoint: undefined
}

This is how the partition shows in mount:

/dev/sda3 on / type ext4 (rw,relatime,errors=remount-ro,data=ordered)

[nebulon]

@fbartels thanks for trying, this is indeed strange, if you change appsdata to say platformdata or boxdata in the path, do you see the same issue? I am asking since the code before checking appsdata is exactly the same with just those other folders being next to appsdata

[fbartels]

Hi @nebulon,

yes the output is the same for those other folders as well.

[girish]

@fbartels the df module works by parsing the output of df -kPT. Can you give us that output? Looks like it's not able to parse it - https://github.com/sindresorhus/df/blob/main/index.js#L21 is the parser, in case you see something obviously wrong. I am guessing maybe this is something locale related. Maybe the ordering of columns is different or something.

[fbartels]

Hi @girish,

hm.. nothing obviously wrong in the output. But since I am running low on disk I think I will anyways order a new system with a bigger disk and will migrate to the new system over the weekend.

root@my:~# df -kPT
Dateisystem    Typ      1024-Blöcke  Benutzt Verfügbar Kapazität Eingehängt auf
udev           devtmpfs     4066396        0   4066396        0% /dev
tmpfs          tmpfs         817672     2768    814904        1% /run
/dev/sda3      ext4        78503608 67520900   6971900       91% /
tmpfs          tmpfs        4088340        0   4088340        0% /dev/shm
tmpfs          tmpfs           5120        0      5120        0% /run/lock
tmpfs          tmpfs        4088340        0   4088340        0% /sys/fs/cgroup
overlay        overlay     78503608 67520900   6971900       91% /var/lib/docker/overlay2/4ae7f0eb3415added8c7214ca91cc07d1e72adca25f3f0180664c4519b67aeb9/merged
overlay        overlay     78503608 67520900   6971900       91% /var/lib/docker/overlay2/a1c60ee0c030fa9581d132dfe0670ee205cd063580b7601861bed09111cc98d5/merged
overlay        overlay     78503608 67520900   6971900       91% /var/lib/docker/overlay2/3420de6abe099a0fc39f5e456a683741feb1f35f05b8c271c50bd673c0e9c889/merged
overlay        overlay     78503608 67520900   6971900       91% /var/lib/docker/overlay2/9e3f8c1df80e792e590a001ce69f2943504d45d9ff5387be9b218857889ca724/merged
overlay        overlay     78503608 67520900   6971900       91% /var/lib/docker/overlay2/e0ab8a0ad5ca7369767c56127979a9c1a42ce3b5c72bde2977fd4fd1a208f347/merged
overlay        overlay     78503608 67520900   6971900       91% /var/lib/docker/overlay2/ab977db2e06989d19d8d954e44613885835ae9ea3e273961654cdcaea6d092e0/merged
overlay        overlay     78503608 67520900   6971900       91% /var/lib/docker/overlay2/a5c29cc69cfbb7db5c18be682e99ce9685e2176eda84284e0707a12927016fe2/merged
overlay        overlay     78503608 67520900   6971900       91% /var/lib/docker/overlay2/700a0718c27054a5b5a223dfe55b5f64f2005624144a26723f982800cbd35e7b/merged
overlay        overlay     78503608 67520900   6971900       91% /var/lib/docker/overlay2/d86d5deb9d18fc86b9af3a752d021c11d3332ee5e6ebb934e414285ffc37c8ef/merged
overlay        overlay     78503608 67520900   6971900       91% /var/lib/docker/overlay2/a62fc2b553be06f863167231bfca52b911be45a96265e3a6ffcf08fbeb19f950/merged
overlay        overlay     78503608 67520900   6971900       91% /var/lib/docker/overlay2/f2625ce2b75792adc5fc867244580148a506428e13786e076ac0d44d7dad30be/merged
overlay        overlay     78503608 67520900   6971900       91% /var/lib/docker/overlay2/f30405f144134254dbeb08e010b6cbd86cfec924278de28a28869e362b99e2b9/merged
overlay        overlay     78503608 67520900   6971900       91% /var/lib/docker/overlay2/accd715cf4cea663e5ea7db5f9785222f48f75af7ab20b7bd2bef6bd1fc31e09/merged
overlay        overlay     78503608 67520900   6971900       91% /var/lib/docker/overlay2/68963526029c37ea6f16a86b9eff6ee739a62474bba40b78591d19b9c0e295d3/merged
overlay        overlay     78503608 67520900   6971900       91% /var/lib/docker/overlay2/954427a9a4f100ebc5661050c6f7d78a666ad2c2425bf9e5893ce45f03c99852/merged
overlay        overlay     78503608 67520900   6971900       91% /var/lib/docker/overlay2/d8959ae21c198714672b34e7d7276f4555df894af0ad8df0e794e86323802188/merged
overlay        overlay     78503608 67520900   6971900       91% /var/lib/docker/overlay2/e0627f3f5cd43b956486eff5491c97dba1515f1850ef6b4dadc9b3bf0948df83/merged
overlay        overlay     78503608 67520900   6971900       91% /var/lib/docker/overlay2/bcf40162a2a3fd30544b4b748c68efc434b3fe0529f7c936a50d77d81c7182ec/merged
overlay        overlay     78503608 67520900   6971900       91% /var/lib/docker/overlay2/1c0010583eb3d2405737feb3c69b0b7e1b7963f5cb4bbaefba442c487e5891c6/merged
overlay        overlay     78503608 67520900   6971900       91% /var/lib/docker/overlay2/c3604832debdffd0ce05cb537a36dfcd3540afca0fe39e7bc860c58a23c3c87d/merged
overlay        overlay     78503608 67520900   6971900       91% /var/lib/docker/overlay2/514be3fe31ac14fb45b54b444914fdc5e5a12838aee72a8d0611207edd0d557b/merged
overlay        overlay     78503608 67520900   6971900       91% /var/lib/docker/overlay2/8f06effb506c1dd1ea3bfba5f5d203c6ca11d1b4bc0ebf5bff3ba6e8c2d38904/merged
overlay        overlay     78503608 67520900   6971900       91% /var/lib/docker/overlay2/8c23d4fba5c58e478d41fabc0ad98e70a568e5829197563bfbdcbe5a1461555f/merged
overlay        overlay     78503608 67520900   6971900       91% /var/lib/docker/overlay2/99995868d541d1c6db64bc548e3b43e798af4b7e3b6c4bfe5fde44cc24ef6595/merged
overlay        overlay     78503608 67520900   6971900       91% /var/lib/docker/overlay2/3f85d3f6f17e0f3dbfa3c414ca63328f082c4b0a462bbb792177bb2fc8e3fcf5/merged
overlay        overlay     78503608 67520900   6971900       91% /var/lib/docker/overlay2/2386e899e129f30a119ffbf3e7307a8ace44d3598e15420565777e50d5f3a02d/merged
overlay        overlay     78503608 67520900   6971900       91% /var/lib/docker/overlay2/bbe9567f7508aedc051b0ba02cd1716861225a209d521f23704002b1e5f31645/merged
overlay        overlay     78503608 67520900   6971900       91% /var/lib/docker/overlay2/8a928eeed78162803163b9eaebe04f1a76406771d733cce891d7b8a5b3d936c4/merged
overlay        overlay     78503608 67520900   6971900       91% /var/lib/docker/overlay2/49d75251c300235c81125ea61d02a911e000a000b3a88c91fc21ef5700ca83de/merged
overlay        overlay     78503608 67520900   6971900       91% /var/lib/docker/overlay2/c3acea88a843e1566054da1fedaf7c37cf2297d9eec9988dc508a0ce6872f17f/merged
overlay        overlay     78503608 67520900   6971900       91% /var/lib/docker/overlay2/b13bc7d2e4fe83e75886765262356b5ac6b7e8ff239c38de8347d17a3903b9a2/merged
overlay        overlay     78503608 67520900   6971900       91% /var/lib/docker/overlay2/875c83993f4812b5d9f2b0bb30fad399a5adf4ffb0aeb015ac4e8aa1ac97bbaf/merged
tmpfs          tmpfs         817672        0    817672        0% /run/user/0
```

[nebulon]

@fbartels hm that node module tries to calculate column boundraries from the header line, which may be causing issues with the longer German words here possibly overflowing the stdout "terminal" rendering and spaces (https://github.com/sindresorhus/df/pull/14)

I guess the real fix would be to force the df exec with an latin1_en or so locale...

[girish]

Indeed, I can reproduce by setting LANG=de_DE.UTF-8

root@my:/home/yellowtent/box# node
Welcome to Node.js v14.15.4.
Type ".help" for more information.
> df = require('@sindresorhus/df')
<ref *1> [AsyncFunction: df] {
  fs: [AsyncFunction (anonymous)],
  file: [AsyncFunction (anonymous)],
  default: [Circular *1]
}
> df.file('/home/yellowtent/appsdata').then(x => console.log(x))
Promise { <pending> }
> {
  filesystem: undefined,
  type: undefined,
  size: NaN,
  used: NaN,
  available: NaN,
  capacity: NaN,
  mountpoint: undefined
}

[girish]

@fbartels Can you please try this.

• Edit /etc/systemd/system/box.service and add LC_ALL to the Environment key.

Environment="HOME=/home/yellowtent" "USER=yellowtent" "DEBUG=box:*,connect-lastmile,-box:ldap" "BOX_ENV=cloudron" "NODE_ENV=production" "LC_ALL=C"

• systemctl daemon-reload
• systemctl restart box

That should hopefully fix the parser.

[fbartels]

Hi @girish ,

thanks that indeed makes the node error go away. Afterwards however i am running into the Graphite/Collectd error already reported in https://forum.cloudron.io/topic/4705/graphite-tagdb-error-tagging-collectd/ and https://forum.cloudron.io/topic/4671/graphite-stuck-starting/.

[girish]

@fbartels awesome, thanks for testing. I am fixing the graphite issue, that error is something else. Fixed in https://git.cloudron.io/cloudron/box/-/commit/f7bd47888ac157fc5b86021f6458b6a7f0d3fba8