Cloudron Forum

Actually I apologize, perhaps my testing was incomplete. I'm not sure what I did wrong but I reverted the live site that had auto-updated, and then I cloned the backup of 3.7.0 with OIDC to a staging domain. I then created a new user in the staging domain as if I was a new member signing up, and then I was able to login in after logging out. I think the issue was with my admin account somehow being unable to login and I haven't quite figured that part out yet. But it seems like it's working fine for local accounts I guess with the new 3.7.0 app version, so that's good news then.

I can't even get the normal Wordpress login form to work for local accounts since the update. I have a client site that has around 50+ members, but this seems to be broken for that particular use-case. If I login with a local user (not OIDC), it seems to constantly fail. How do we work around this?

For now, I've reverted from a backup and am about to stage this update for testing purposes. Is there a way around this? basically the use-case is they have a site which has a detailed form that members have to fill out to get access. Once they fill out the form they can login with the credentials. This all worked perfectly until the latest 3.7.0 Cloudron version update for the WordPress Developer app. I love the Cloudron OIDC login method but it should be optional and not the required method for local app accounts. Am I missing something? I have to assume this functionality still exists for this use-case so local users can login, but I can't seem to get it to work on 3.7.0.

FWIW, I uninstalled the app so that the image would be removed, confirmed it was removed, and then re-installed. The image size changed slightly from 6.57 GB to 6.51 GB, but still quite large. I may be doing in the wrong area though if we don't think this is a huge concern, just really strange to me why it's so much larger than all the other images when it's supposedly a fairly lean app.

For the life of me I cannot seem to reclaim the over 2 GB of disk space from images in Docker, I am puzzled as to why that is happening.

@girish does that mean the image should only be about 1.4 GB in size rather than the 6+GB I’m seeing currently? Very odd if that’s the case, I wonder how that exploded in size.

I'm curious why the Umami app is so massive. It's such a lightweight application... is it expected to be that large? I wonder if this is somehow a contributor to the issue of reclaimable space but not actually being removed since it's still active? Just throwing against a wall though, not sure if that's valid. haha.

Here is my docker image list by the way in case this points to any issues at all:

$ sudo docker image ls
REPOSITORY                                     TAG                         IMAGE ID       CREATED        SIZE
cloudron/org.nodebb.cloudronapp                20240403-154317-254cac2fd   3b7a24812a6e   32 hours ago   2.86GB
cloudron/com.invoiceninja.cloudronapp2         20240401-092303-3942cee63   dcda0788296b   3 days ago     3.6GB
cloudron/io.gitea.cloudronapp                  20240326-072227-110a2e6cd   7ed238e459fa   9 days ago     2.74GB
cloudron/sh.ntfy.cloudronapp                   20240326-072218-802b89e89   807661895891   9 days ago     2.26GB
cloudron/org.wordpress.unmanaged.cloudronapp   20240319-202918-2976247a8   1117a989fc3a   2 weeks ago    2.3GB
cloudron/org.radicale.cloudronapp2             20240319-154937-11289c349   036b57520b5b   2 weeks ago    2.22GB
registry.docker.com/cloudron/postgresql        5.2.1                       333f887a27f7   3 weeks ago    2.75GB
cloudron/org.piwik.cloudronapp                 20240308-102528-2182681db   3ddaa6276ff3   3 weeks ago    2.51GB
cloudron/is.umami.cloudronapp                  20240307-081949-7105f94d1   d0512bd1a4c1   4 weeks ago    6.57GB
registry.docker.com/cloudron/sftp              3.8.6                       b735f2120189   4 weeks ago    2.23GB
cloudron/com.github.bitwardenrs                20240303-104654-927b1cf62   1393b91919fa   4 weeks ago    3.51GB
registry.docker.com/cloudron/mail              3.12.1                      ea18fc4dd1c7   5 weeks ago    2.96GB
registry.docker.com/cloudron/mongodb           6.0.0                       4b95d24318a2   8 weeks ago    2.69GB
cloudron/net.roundcube.cloudronapp             20240121-133422-3162a79c7   29ad5d8091ed   2 months ago   2.23GB
cloudron/louislam.uptimekuma.app               20240102-093304-840efe2c0   0e7aea4082d9   3 months ago   3.29GB
cloudron/tech.ittools.cloudron                 20231221-163307-91643bd95   961340a3d920   3 months ago   2.22GB
cloudron/io.cloudron.surfer                    20231216-181458-705d2061b   8a2725a40c45   3 months ago   2.39GB
registry.docker.com/cloudron/graphite          3.4.3                       dbd026164ada   5 months ago   2.28GB
cloudron/net.jirafeau.cloudronapp              20231013-024132-1436ee8da   4155ebdab88f   5 months ago   2.21GB
registry.docker.com/cloudron/redis             3.5.2                       80e7a4079e6b   6 months ago   2.22GB
registry.docker.com/cloudron/mysql             3.4.2                       c7085a52532b   6 months ago   2.53GB
registry.docker.com/cloudron/turn              1.7.2                       152b1fb9690e   6 months ago   2.22GB

@girish That worked well for the volumes, it cleared up the 1.666GB of storage space. Thank you!

What about the images though? It says over 2 GB is reclaimable but it doesn't seem like it is from what I can tell. If I tried to do a similar command specific to images such as docker image prune -a it still gave me a 0 byte response showing it cleaned up nothing. Am I missing something when it comes to reclaimable images in Docker?

I'm trying to understand how I can clean up the disk by lowering the disk usage. I applied docker image prune -a but it reclaimed all of 0 bytes, lol, so it didn't seem to work for me in the way I was hoping.

The reason for this is I'm using the Public Cloud instance of OVH but specifically their "flexible" cloud instance so that the instance can be sized up and down for CPU and memory rather than only up. The downside is it gives only a 50 GB hard disk on a flexible instance which is fine with me though as block storage is pretty inexpensive. So I have boxdata and appsdata on different external block storage disks with plenty of space.

But the main disk is still quite consumed by Docker. Here's my current usage for the main disk:

/dev/sda1 mounted at /

41.1 GB used of 51.84 GB
Speed: 1180 MB/sec
This disk contains:

  docker 14.37 GB
  docker-volumes 5.51 GB
  platformdata 5.48 GB
  /apps.swap 4.29 GB
  Everything else (Ubuntu, etc) 11.45 GB

I have considered reducing the swap if possible since I have ample memory, but I'd rather keep it if needed.

The docker is over 14 GB, and docker-volumes just over 5 GB. Basically the main "Docker" tag is the main user and to some degree I think that makes sense as it's the main platform, but I'm curious if it should be that high still. I noticed another post where a user tried the docker image prune -a and reclaimed nearly 10 GB to squeeze it down to closer to 4 GB but that didn't do anything for me unfortunately.

sudo docker image prune -a
WARNING! This will remove all images without at least one container associated to them.
Are you sure you want to continue? [y/N] y
Deleted Images:
untagged: registry.docker.com/cloudron/base:4.2.0
untagged: registry.docker.com/cloudron/base@sha256:46da2fffb36353ef714f97ae8e962bd2c212ca091108d768ba473078319a47f4
deleted: sha256:6ec7c1ab3983e1ebdbf75ced0e8089db363083aa4ed004be0022d45bafaef4f9

Total reclaimed space: 0B

It looks like the images are the main thing when I run the docker system df command (accounts for the full docker stat on the Cloudron disk usage chart):

sudo docker system df
TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          22        22        14.37GB   2.21GB (15%)
Containers      87        64        0B        0B
Local Volumes   162       134       5.537GB   1.667GB (30%)
Build Cache     0         0         0B        0B

What's odd is that it shows nearly 3+ GB of reclaimable space, but it seems like that isn't actually recoverable to add back to available disk space, at least not in any way that I've found so far.

Is there anything I can do here to help conserve space, is there perhaps any commands I can run to clean things up further for Docker? Any guidance would be appreciated. If needed I may just get a small external block storage for the platform data but I am hoping to avoid that so I can keep all the DB stuff on the main higher-performing disk.

Genuinely curious how people scan their machines for malicious files. With many major recent WordPress vulnerabilities that were knowingly exploited, it’s made me think a bit more on how to monitor for such things. Of course there are tools within the WordPress ecosystem for example for monitoring things and I utilize these already to an extent, however, those are certainly not infallible so they’re usually best combined with a scanning tool at the server level as well from what I’ve been reading.

I was wondering if anybody is implemented something like this and what tools they might recommend. It seems like a popular one is ClamAV, but not sure how this may impact server performance (I guess I’ll just have to try). Any other recommendations that people can vouch for? I figured it doen’t even need to scan the whole server but just the boxdata folders for example which may help on minimizing any possible performance impact.

@marcusquinn - yes, a huge vote for that. I think this was referenced before in a feature request too if I recall correctly. It is very annoying when I need to quickly restart an app and my system backup happens to be running preventing me from doing a quick task. I usually just get around this by backing up in the middle of the night but sometimes I want to also do a system backup in the afternoon but on my system it takes nearly 1.5 hours for a backup to complete so then in that time I’m forbidden from restarting an app for example which is quite annoying. Would love to see multi tasking for that.

I don't think NodeBB supports this just yet (not that I could find), but it'd be awesome when it does arrive if we could enable Passkey support in this forum. Discourse added it and the forums I participate in have enabled it, it's pretty slick.

That makes sense, not sure why I hadn't thought of that one. I temporarily moved the invoice Ninja app to a different subdomain, created a quick LAMP app on the original subdomain with the file I needed, then verified and undid it all. Thank you for the idea @robi.

This may be a strange request but I have the need to temporarily upload a txt file to the public directory of Invoice Ninja for verification purposes, however I cannot seem to get this to work no matter where I put the txt (text) file. If I add to public and try to access it I still get a 404. Is this possible with the way this app is packaged?

Good to know, thanks Girish. It never happened again, only once so I guess it was transient in nature.

I noticed a backup failed last night, it's the first failure I've seen with this type of error, and the very next backup succeeded so I just wanted to inquire about this one. The last bit of the logs are below for reference. The main error portion seemed to be InvalidArgument InvalidArgument: Range specified is not valid for source object.

Jan 20 04:18:37box:tasks update 23477: {"percent":53.63157894736845,"message":"Copying with concurrency of 10"}
Jan 20 04:18:37box:tasks update 23477: {"percent":53.63157894736845,"message":"Copying files from 0-1"}
Jan 20 04:18:37box:tasks update 23477: {"percent":53.63157894736845,"message":"Copying (multipart) {CloudronHostname}/snapshot/app_6b3ebc4f-9708-4243-bc21-4c9a01bc3499.tar.gz.enc"}
Jan 20 04:18:40box:tasks update 23477: {"percent":53.63157894736845,"message":"Copying part 1 - /cloudron-backups/{CloudronHostname}/snapshot/app_6b3ebc4f-9708-4243-bc21-4c9a01bc3499.tar.gz.enc bytes=0-1073741823"}
Jan 20 04:18:40box:tasks update 23477: {"percent":53.63157894736845,"message":"Copying part 3 - /cloudron-backups/{CloudronHostname}/snapshot/app_6b3ebc4f-9708-4243-bc21-4c9a01bc3499.tar.gz.enc bytes=2147483648-3221225471"}
Jan 20 04:18:54box:tasks update 23477: {"percent":53.63157894736845,"message":"Copying part 3 - Etag: \"4e4b54bc08223e06623dd522542dddb4\""}
Jan 20 04:18:54box:tasks update 23477: {"percent":53.63157894736845,"message":"Copying part 4 - /cloudron-backups/{CloudronHostname}/snapshot/app_6b3ebc4f-9708-4243-bc21-4c9a01bc3499.tar.gz.enc bytes=3221225472-3909745923"}
Jan 20 04:18:54box:tasks update 23477: {"percent":53.63157894736845,"message":"Aborting multipart copy of {CloudronHostname}/snapshot/app_6b3ebc4f-9708-4243-bc21-4c9a01bc3499.tar.gz.enc"}
Jan 20 04:18:54box:tasks update 23477: {"percent":53.63157894736845,"message":"Retrying (1) multipart copy of {CloudronHostname}/snapshot/app_6b3ebc4f-9708-4243-bc21-4c9a01bc3499.tar.gz.enc. Error: InvalidArgument: Range specified is not valid for source object 400"}
Jan 20 04:18:54box:storage/s3 copy: s3 copy error when copying {CloudronHostname}/snapshot/app_6b3ebc4f-9708-4243-bc21-4c9a01bc3499.tar.gz.enc: InvalidArgument: Range specified is not valid for source object
Jan 20 04:18:54box:backuptask copy: copied to 2024-01-20-120001-496/app_{appHostname}_v3.4.0 errored. error: Error copying {CloudronHostname}/snapshot/app_6b3ebc4f-9708-4243-bc21-4c9a01bc3499.tar.gz.enc (3909745924 bytes): InvalidArgument InvalidArgument: Range specified is not valid for source object
Jan 20 04:18:54box:tasks update 23477: {"percent":53.63157894736845,"message":"Copied 1 files with error: BoxError: Error copying {CloudronHostname}/snapshot/app_6b3ebc4f-9708-4243-bc21-4c9a01bc3499.tar.gz.enc (3909745924 bytes): InvalidArgument InvalidArgument: Range specified is not valid for source object"}
Jan 20 04:18:54box:tasks setCompleted - 23477: {"result":null,"error":{"stack":"BoxError: Error copying {CloudronHostname}/snapshot/app_6b3ebc4f-9708-4243-bc21-4c9a01bc3499.tar.gz.enc (3909745924 bytes): InvalidArgument InvalidArgument: Range specified is not valid for source object\n at done (/home/yellowtent/box/src/storage/s3.js:338:48)\n at Response.<anonymous> (/home/yellowtent/box/src/storage/s3.js:414:71)\n at Request.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:367:18)\n at Request.callListeners (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:106:20)\n at Request.emit (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:78:10)\n at Request.emit (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:686:14)\n at Request.transition (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:22:10)\n at AcceptorStateMachine.runTo (/home/yellowtent/box/node_modules/aws-sdk/lib/state_machine.js:14:12)\n at /home/yellowtent/box/node_modules/aws-sdk/lib/state_machine.js:26:10\n at Request.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:38:9)","name":"BoxError","reason":"External Error","details":{},"message":"Error copying {CloudronHostname}/snapshot/app_6b3ebc4f-9708-4243-bc21-4c9a01bc3499.tar.gz.enc (3909745924 bytes): InvalidArgument InvalidArgument: Range specified is not valid for source object"}}
Jan 20 04:18:54box:taskworker Task took 1133.43 seconds
Jan 20 04:18:54box:tasks update 23477: {"percent":100,"result":null,"error":{"stack":"BoxError: Error copying {CloudronHostname}/snapshot/app_6b3ebc4f-9708-4243-bc21-4c9a01bc3499.tar.gz.enc (3909745924 bytes): InvalidArgument InvalidArgument: Range specified is not valid for source object\n at done (/home/yellowtent/box/src/storage/s3.js:338:48)\n at Response.<anonymous> (/home/yellowtent/box/src/storage/s3.js:414:71)\n at Request.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:367:18)\n at Request.callListeners (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:106:20)\n at Request.emit (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:78:10)\n at Request.emit (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:686:14)\n at Request.transition (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:22:10)\n at AcceptorStateMachine.runTo (/home/yellowtent/box/node_modules/aws-sdk/lib/state_machine.js:14:12)\n at /home/yellowtent/box/node_modules/aws-sdk/lib/state_machine.js:26:10\n at Request.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:38:9)","name":"BoxError","reason":"External Error","details":{},"message":"Error copying {CloudronHostname}/snapshot/app_6b3ebc4f-9708-4243-bc21-4c9a01bc3499.tar.gz.enc (3909745924 bytes): InvalidArgument InvalidArgument: Range specified is not valid for source object"}}
Error copying {CloudronHostname}/snapshot/app_6b3ebc4f-9708-4243-bc21-4c9a01bc3499.tar.gz.enc (3909745924 bytes): InvalidArgument InvalidArgument: Range specified is not valid for source object

Well I'm kind of ashamed to say it but I gave up, I spent far too many hours on this one (felt like ~25 between the last three days) just on this one single issue in order to avoid an IP block for my clients as Microsoft was blocking the IP of the host machine, and I still couldn't get it to work. I eventually opted instead to create a new Public Cloud instance at OVH instead, migrated all the data to it, using the floating IP, and things seem to be working fine now (only accessible with one public IP address so Docker had no other choice, lol). Sorry there's no confirmed solution to this one.

It looks like they're all empty honestly at 4K file sizes and when I cat a couple of them they're all with the same content "empty".

ubuntu@ns574752:~$ ls -alh /sys/devices/virtual/dmi/id/
total 0
drwxr-xr-x 3 root root    0 Jan 16 17:43 .
drwxr-xr-x 3 root root    0 Jan 16 17:43 ..
-r--r--r-- 1 root root 4.0K Jan 16 17:57 bios_date
-r--r--r-- 1 root root 4.0K Jan 16 17:57 bios_release
-r--r--r-- 1 root root 4.0K Jan 16 17:43 bios_vendor
-r--r--r-- 1 root root 4.0K Jan 16 17:57 bios_version
-r--r--r-- 1 root root 4.0K Jan 16 17:57 board_asset_tag
-r--r--r-- 1 root root 4.0K Jan 16 17:43 board_name
-r-------- 1 root root 4.0K Jan 16 17:57 board_serial
-r--r--r-- 1 root root 4.0K Jan 16 17:43 board_vendor
-r--r--r-- 1 root root 4.0K Jan 16 17:57 board_version
-r--r--r-- 1 root root 4.0K Jan 16 17:43 chassis_asset_tag
-r-------- 1 root root 4.0K Jan 16 17:57 chassis_serial
-r--r--r-- 1 root root 4.0K Jan 16 17:57 chassis_type
-r--r--r-- 1 root root 4.0K Jan 16 17:57 chassis_vendor
-r--r--r-- 1 root root 4.0K Jan 16 17:57 chassis_version
-r--r--r-- 1 root root 4.0K Jan 16 17:43 modalias
drwxr-xr-x 2 root root    0 Jan 16 17:57 power
-r--r--r-- 1 root root 4.0K Jan 16 17:57 product_family
-r--r--r-- 1 root root 4.0K Jan 16 17:43 product_name
-r-------- 1 root root 4.0K Jan 16 17:43 product_serial
-r--r--r-- 1 root root 4.0K Jan 16 17:57 product_sku
-r-------- 1 root root 4.0K Jan 16 17:43 product_uuid
-r--r--r-- 1 root root 4.0K Jan 16 17:43 product_version
lrwxrwxrwx 1 root root    0 Jan 16 17:43 subsystem -> ../../../../class/dmi
-r--r--r-- 1 root root 4.0K Jan 16 17:43 sys_vendor
-rw-r--r-- 1 root root 4.0K Jan 16 17:43 uevent

ubuntu@ns574752:~$ sudo cat /sys/devices/virtual/dmi/id/product_family
empty

@Kubernetes That sounds like something similar to what ChatGTP told me too so that's good to know. It's strange, because I have what I assumed to be the correct rules in place, as I can run a command curl ifconfig.me and it will now show me the correct IP, but this is for the host, it still uses the unwanted IP if I am running that from a container. So it seems like I need to do something specific to Docker.

@robi said in SMTP using wrong IP address on interface with multiple addresses:

The metric param also has a weight function, lower being higher priority.

Ah yes, I added different weights to mine and still unfortunately see no difference in behaviour. It works fine for host traffic, just not Docker traffic.

@girish said in SMTP using wrong IP address on interface with multiple addresses:

Also, is the Floating IP like DigitalOcean Floating IP? DO floating IP cannot be used as address for outbound traffic , it is only used for incoming traffic. This is because the IP is not assigned to any interface, if I recall correctly.

I don't think that's accurate anymore, as I actually stumbled across this (in part trying to see if I could learn anything from it to apply to OVH, lol): https://docs.digitalocean.com/products/networking/reserved-ips/how-to/outbound-traffic/ - unless "Reserved IP" is different than "Floating IP" for DigitalOcean? Also I found this URL which I was using (it got me in the right direction as my host traffic now goes properly out the floating IP, but just not my Docker traffic yet) which is showing how to use the OVH Floating IP as an outbound source IP: https://medium.com/@PHaroZ_/use-ovh-floating-ip-for-outgoing-traffic-including-from-containers-cd03edaecc3

I'm somehow stuck at trying to get Docker to follow the same rules as my host is following.

Hmm, that rings a bell but I don't think I've seen that for a few years now. The interface (doing a ip a command) shows this:

2: enp5s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether {macAddress} brd ff:ff:ff:ff:ff:ff
    inet {floatingIP} scope global enp5s0f0
       valid_lft forever preferred_lft forever
    inet {autoAssignedIP} metric 100 brd {gatewayIP} scope global dynamic enp5s0f0
       valid_lft 72471sec preferred_lft 72471sec
    inet6 {autoAssignedIPv6} scope link 
       valid_lft forever preferred_lft forever

I assumed too since the floating IP is listed first that it'd be the preferred one, but doesn't seem to be. In the past, I'm pretty sure that's what happened, as long as I added the floating IP higher in the list than the auto-assigned IP, Cloudron's Docker containers would automatically pick up the floating IP.

@girish - is there a way to define which IP address is used as the source when sending outbound traffic (i.e. mail)? I'm really struggling with this. My gut tells me this is coming from an OVH DHCP issue, but I've at least gotten the host to send all traffic outbound with a source of my floating IP, however it's now the Cloudron Docker containers that don't seem to be adhering to that, which is where I'm sure OVH Support will point back to Cloudron at that point. So any help on this would be appreciated.