Cloudron Forum

One of our Nextcloud installs broke in package update 4.21.* , with all users files not being visible in the UI. Also the external mount (A Linode object store using the S3 interface) gave errors when trying to access it. These are the updates that moved from Nextcloud 27 to Nextcloud 28. The disk usage shows the same amount of usage as before, so I expect the files are still present in the container.

Going back to the last backup of version 4.20.6 update restored all the files, so everything is currently "fine", but it'd be nice to know why it broke.

Interestingly, a newer Nextcloud install on Cloudron has updated without issue.

So I'm guessing some deprecated functionality is still being used by our install, from possibly years ago, and it was removed in this update. However, I don't have any idea what it was, or if it was part of the Cloudron packaging or part of the Nextcloud update. Nextcloud's changelogs are not great to read, and I haven't found any kind of migration guide for Nextcloud 28.

Does anyone who is familiar with Nextcloud have any ideas where I should start looking?

Is there, or is there planned to be, a way to restrict an OpenID Connect provider to members of a certain group?

Specifically, we have a web site hosted outside of Cloudron, that it would be great to use our Cloudron accounts to control user access to. OpenID Connect (and other OAuth variants) are supported, so Cloudron just coming out with a Provider, is promising. But we want to restrict it to a particular group of users.

@girish said in Backups failing with ERR_STREAM_WRITE_AFTER_END:

npm install git+https://github.com/cloudron-io/aws-sdk-js.git#continue_once

This has worked for me.

Just as a note, I had to su yellowtent rather than sudo, as NPM kept complaining about root owned cache files (since HOME was still pointing at /root/). Just in case anyone else needs to follow these instructions.

@girish Ah, great - I'm one of the comments on that thread as well. That's an interesting problem.

For the last 5 days, every backup is failing with ERR_STREAM_WRITE_AFTER_END. We are using the Linode backend, and are in communication with them as well to find out if anything changed on their end.

(stdout): 2021-03-21T06:14:46.894Z box:storage/s3 Error uploading [cloudron/snapshot/app_0443482b-6d44-49a5-b02c-92412307a711.tar.gz.enc]: s3 upload error. Error [ERR_STREAM_WRITE_AFTER_END]: write after end
at writeAfterEnd (_http_outgoing.js:668:15)
at ClientRequest.end (_http_outgoing.js:788:7)
at features.constructor.writeBody (/home/yellowtent/box/node_modules/aws-sdk/lib/http/node.js:137:14)
at ClientRequest.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/http/node.js:102:14)
at ClientRequest.emit (events.js:315:20)
at ClientRequest.EventEmitter.emit (domain.js:467:12)
at HTTPParser.parserOnIncomingClient [as onIncoming] (_http_client.js:606:11)
at HTTPParser.parserOnHeadersComplete (_http_common.js:126:17)
at TLSSocket.socketOnData (_http_client.js:509:22)
at TLSSocket.emit (events.js:315:20) {
code: 'NetworkingError',
region: 'us-east-1',
hostname: '[redacted].us-east-1.linodeobjects.com',
retryable: true,
time: 2021-03-21T06:14:46.873Z

Based on my diagnosis, this is caused by the "worker" scheduler/cron task. Every time that runs (based on the log), I see a new file in /app/code/tmp/cache owned by root. Sometimes this new file is in a new directory (as the cache splits the storage across multiple directories on demand based on the random file name). When this newly created root owned folder is used by the normal process (running as www-data), it fails the permission check and users see the error message.

I've been "fixing" this by manually running

/app/code/tmp/cache# find -user root -exec chown www-data:www-data {} \;

at the terminal.

@girish Yes I did, and the problem with the certificates is now fixed. Thank you!

@girish Actually, I just double-checked and the update didn't install. I'd seen it was in the process of installing, then had finished, so had assumed I was on 5.5, but I'm still using 5.4 . I've retriggered the update process and will test again if it finishes.

Edit: Hmm, it ran and again failed, but I refreshed the page before clicking to get the logs, and the nightly scheduled update had started! The displayed message was something like "failed with signal null".

Edit: Ah, I was able to grab the log (and the log of cloudton-updater) and found the problem. A little while ago, I'd started installing a tool used by my hosting provider, without realizing it was going to trigger an apt update etc, which Cloudron specifically warns against. I broke out of it, but it seems I left dpkg with unconfigured packages. I've fixed that now, and am trying the update again.

@girish I'm still getting the same behaviour, and having to reapply the manual change whenever a app updates.

Great, I'm looking forward to it.
So you know, the old certificate got put back into place and I had to re-apply the manual change.
Do you know what circumstances cause the cert to be reevaluated? For example, adding applications? Or is it just something that will happen on a regular schedule?

@girish Yes, that was the case.

@girish Thank you, the workaround worked.

I grabbed out the certs that were there before. I don't know if it would help to attach them, but they look like standard Lets Encrypt Authority X3 issued certs.

@girish OK, I've just tried that (for the main domain), but I'm not seeing any change.

This is affecting both incoming SMTP (so is domain agnostic at that point), as well as IMAP (which I assume also uses STARTTLS or equivalent before using a specific domain login).

If it's helpful, I'm familiar enough with sysadmin to be able to access the docker container command line and such, to get more information/apply changes.

Edit: I also just tried cycling the mail service, in case the change hadn't applied yet.

Edit: I also just tried disabling email across all domains (which did disable the mail service, based on the SSL test), but as soon as I re-enabled it for the main domain, the same error occurred.

We're using custom wildcard certificates for all our domains. When we made this switch, email didn't change to using the new certificate, and kept using the Lets Encrypt cert.

This cert has now expired. I've found various issues in the past similar to this, and tries those fixes (add new domain then remove it, restart email service), but the issue persists.

This appears to be a case of https://forum.cloudron.io/post/6533 . Applying the linked setting change fixed the issue.

For some of our users, their SMTP connections are being denied, with:

902 Host names have more than one DNS label

This error appears to come from https://codeclimate.com/github/msimerson/Haraka/plugins/helo.checks.js/source

I don't know how to walk them through finding out what HELO their device is sending to validate it, but it seems related to the plugin.cfg.reject.valid_hostname setting. Did this change in 5.0.3 ?

@girish As far as I know, it's correct - unless DKIM has the IP included, since that did change during the migration. Unfortunately, the Email section of Cloudron wasn't properly giving the needed information - previously it was saying DKIM was incorrect, but the display just said something like "expected null to equal null". DKIM isn't showing in the Email status page at all with the workaround I added here.

I'll put that in as a separate issue, though, once the update comes out and I see the proper message.

Last night our Cloudron updated to 4.4.0, and got stuck in a restart loop.
The error in box.log was:

/home/yellowtent/box/src/mail.js:524
                    if (!record.status) message.push(`${type.toUpperCase()} DNS record (${record.type}) did not match.\n    * Hostname: \`${record.name}\`\n
   * Expected: \`${record.expected}\`\n    * Actual: \`${record.value}\``);
                                ^

TypeError: Cannot read property 'status' of undefined
    at Object.keys.forEach (/home/yellowtent/box/src/mail.js:524:33)
    at Array.forEach (<anonymous>)
    at /home/yellowtent/box/src/mail.js:522:41
    at /home/yellowtent/box/src/mail.js:503:13
    at /home/yellowtent/box/node_modules/async/dist/async.js:3888:9
    at /home/yellowtent/box/node_modules/async/dist/async.js:473:16
    at iteratorCallback (/home/yellowtent/box/node_modules/async/dist/async.js:1064:13)
    at /home/yellowtent/box/node_modules/async/dist/async.js:969:16
    at /home/yellowtent/box/node_modules/async/dist/async.js:3885:13
    at /home/yellowtent/box/src/mail.js:471:17

I patched around this by editing that file and adding a

if (!record) return;

into the forEach callback.

We've been having issues with Cloudron not correctly reading the mail config (reporting DNS misconfiguration, with error messages about expecting null to equal null), since we migrated to a new host a few weeks ago. This may be related, but I haven't had time to report the issues that happening in that migration yet.

Rocketchat group calls are also using Jitsi, so including that would be a big plus for us.

We are attempting to use a wildcard cert for one of our domains.

I attempted using the key and cert as provided (no ca chain), which the UI accepted, but then on restart, all subdomains were using self-signed certs.

After working around HSTS, I was able to get back into the system and switch back to Lets Encrypt.

Next, I tried including the ca bundle in the cert file. When I did it the wrong way around (ca certs before the wildcard cert), the UI reported an error. When done the other way (as per NGINX documentation), the UI accepted the cert. However, all subdomains were using a self-signed cert again.

The renewcerts log showed an error along the lines of "null certificate, falling back to self-signed". Unfortunately, I didn't capture the logs at the time, and the available history doesn't go back that far.

I've tried various certificate inspection tools, that have reported the certificate as valid. This is the same provider we use for providing wildcard certs for non-Cloudron domains, that we SSL-terminate with NGINX, which is why I thought to try the ca-bundle inclusion.

Can anyone provide any insight into why this would have failed in this manner?