Peertube restarting over and over

ChristopherMag

Here is the output of the logs:

Dec 10 08:10:45 [peertube.cloudron.magnusonfinancial.com:443] 2020-12-10 13:10:45.196 info: Database dbd2fb1510462d45d084b88143184152fc is ready.
Dec 10 08:10:45 [peertube.cloudron.magnusonfinancial.com:443] 2020-12-10 13:10:45.336 info: Using mail:2525 as SMTP server.
Dec 10 08:10:45 [peertube.cloudron.magnusonfinancial.com:443] 2020-12-10 13:10:45.337 info: Testing SMTP server...
Dec 10 08:10:45 [peertube.cloudron.magnusonfinancial.com:443] 2020-12-10 13:10:45.361 info: Connection established to 172.18.0.7:2525 {
Dec 10 08:10:45 "component": "smtp-connection",
Dec 10 08:10:45 "sid": "OdFXvbwRvo4",
Dec 10 08:10:45 "tnx": "network",
Dec 10 08:10:45 "localAddress": "172.18.16.16",
Dec 10 08:10:45 "localPort": 37310,
Dec 10 08:10:45 "remoteAddress": "172.18.0.7",
Dec 10 08:10:45 "remotePort": 2525
Dec 10 08:10:45 }
Dec 10 08:10:46 [peertube.cloudron.magnusonfinancial.com:443] 2020-12-10 13:10:46.367 info: User "peertube.app@cloudron.magnusonfinancial.com" failed to authenticate {
Dec 10 08:10:46 "component": "smtp-connection",
Dec 10 08:10:46 "sid": "OdFXvbwRvo4",
Dec 10 08:10:46 "tnx": "smtp",
Dec 10 08:10:46 "username": "peertube.app@cloudron.magnusonfinancial.com",
Dec 10 08:10:46 "action": "authfail",
Dec 10 08:10:46 "method": "PLAIN"
Dec 10 08:10:46 }
Dec 10 08:10:46 [peertube.cloudron.magnusonfinancial.com:443] 2020-12-10 13:10:46.369 error: Failed to connect to SMTP mail:2525. {
Dec 10 08:10:46 "err": {
Dec 10 08:10:46 "stack": "Error: Invalid login: 535 5.7.8 Authentication failed\n at SMTPConnection._formatError (/app/code/server/node_modules/nodemailer/lib/smtp-connection/index.js:784:19)\n at SMTPConnection._actionAUTHComplete (/app/code/server/node_modules/nodemailer/lib/smtp-connection/index.js:1523:34)\n at SMTPConnection.<anonymous> (/app/code/server/node_modules/nodemailer/lib/smtp-connection/index.js:550:26)\n at SMTPConnection._processResponse (/app/code/server/node_modules/nodemailer/lib/smtp-connection/index.js:942:20)\n at SMTPConnection._onData (/app/code/server/node_modules/nodemailer/lib/smtp-connection/index.js:749:14)\n at Socket.SMTPConnection._onSocketData (/app/code/server/node_modules/nodemailer/lib/smtp-connection/index.js:195:44)\n at Socket.emit (events.js:310:20)\n at addChunk (_stream_readable.js:286:12)\n at readableAddChunk (_stream_readable.js:268:9)\n at Socket.Readable.push (_stream_readable.js:209:10)\n at TCP.onStreamRead (internal/stream_base_commons.js:186:23)",
Dec 10 08:10:46 "message": "Invalid login: 535 5.7.8 Authentication failed",
Dec 10 08:10:46 "code": "EAUTH",
Dec 10 08:10:46 "response": "535 5.7.8 Authentication failed",
Dec 10 08:10:46 "responseCode": 535,
Dec 10 08:10:46 "command": "AUTH PLAIN"
Dec 10 08:10:46 }
Dec 10 08:10:46 }
Dec 10 08:10:46 npm ERR! code ELIFECYCLE
Dec 10 08:10:46 npm ERR! errno 255
Dec 10 08:10:46 npm ERR! peertube@2.4.0 start: `node dist/server`
Dec 10 08:10:46 npm ERR! Exit status 255
Dec 10 08:10:46 npm ERR!
Dec 10 08:10:46 npm ERR! Failed at the peertube@2.4.0 start script.
Dec 10 08:10:46 npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

When I initially installed peertube I had outbound email configured.

Due to a compliance issue I now have outbound email explicitly disabled so that no apps can send email external to the server.

I don't know if I restarted peertube since making that change so this issue may have been one waiting to show up the first time it restarted after having outbound email disabled.

ChristopherMag

@christophermag

After commenting out the smtp and email sections of the production.yaml I now get the following:

Dec 10 08:18:09 ==> Updating Ldap credentials
Dec 10 08:18:09 UPDATE 0
Dec 10 08:18:09 ==> Updating configs
Dec 10 08:18:09 ==> Starting PeerTube
Dec 10 08:18:09
Dec 10 08:18:09 > peertube@2.4.0 start /app/code/server
Dec 10 08:18:09 > node dist/server
Dec 10 08:18:09
Dec 10 08:18:09 internal/fs/utils.js:230
Dec 10 08:18:09 throw err;
Dec 10 08:18:09 ^
Dec 10 08:18:09
Dec 10 08:18:09 Error: ENOENT: no such file or directory, mkdir '/app/code/server/storage/logs'
Dec 10 08:18:09 at Object.mkdirSync (fs.js:841:3)
Dec 10 08:18:09 at Object.module.exports.makeDirSync (/app/code/server/node_modules/fs-extra/lib/mkdirs/make-dir.js:101:15)
Dec 10 08:18:09 at Object.<anonymous> (/app/code/server/dist/server/helpers/logger.js:11:12)
Dec 10 08:18:09 at Module._compile (internal/modules/cjs/loader.js:1156:30)
Dec 10 08:18:09 at Object.Module._extensions..js (internal/modules/cjs/loader.js:1176:10)
Dec 10 08:18:09 at Module.load (internal/modules/cjs/loader.js:1000:32)
Dec 10 08:18:09 at Function.Module._load (internal/modules/cjs/loader.js:899:14)
Dec 10 08:18:09 at Module.require (internal/modules/cjs/loader.js:1042:19)
Dec 10 08:18:09 at require (internal/modules/cjs/helpers.js:77:18)
Dec 10 08:18:09 at Object.<anonymous> (/app/code/server/dist/server/initializers/checker-before-init.js:7:18) {
Dec 10 08:18:09 errno: -2,
Dec 10 08:18:09 syscall: 'mkdir',
Dec 10 08:18:09 code: 'ENOENT',
Dec 10 08:18:09 path: '/app/code/server/storage/logs'
Dec 10 08:18:09 }
Dec 10 08:18:09 npm ERR! code ELIFECYCLE
Dec 10 08:18:09 npm ERR! errno 1
Dec 10 08:18:09 npm ERR! peertube@2.4.0 start: `node dist/server`
Dec 10 08:18:09 npm ERR! Exit status 1
Dec 10 08:18:09 npm ERR!
Dec 10 08:18:09 npm ERR! Failed at the peertube@2.4.0 start script.
Dec 10 08:18:09 npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
Dec 10 08:18:09

ChristopherMag

@christophermag It looks like the 2nd output is due to me corrupting the production.yaml as the start.sh script overwrites parts of it so my manual modifications conflicted.

Currently trying to get back to a properly formatted production.yaml but the first set of logs in the thread are the ones related to the real problem and not the one I caused by modifying production.yaml while not in recovery mode.

ChristopherMag

@christophermag I had to delete an appended section of smtp configuration from the end of the file and uncomment out the original smtp section and now the app tries to start and gets the error messages from the first post.

ChristopherMag

@christophermag I have tried to comment out the section of start.sh for this app that writes the smtp server configuration into the production.yaml so I can confirm that the smtp server not responding is the cause of this issue but start.sh is in a read only file system.

Running start.sh from /app/pkg/ results in an error message as it is looking for a package.json file.

Changing to /app/code/server and then executing /app/pkg/start.sh results in the server trying to start and getting the smtp error messages from the first post.

Will now try to copy start.sh somewhere and see if a modified version that doesn't set smtp server config in the production.yaml along with commenting that config out of production.yaml allows me to start peertube.

nebulon

To take a step back, maybe we can see why you need to modify those files and if so, if there is something missing in the package to make it possible or more stable. I've lost a bit of overview as to what you changed where

ChristopherMag

After copying /app/pkg/start.sh to /app/data/start.sh I commented out the lines that set the smtp server configuration and saved start.sh.

I then commented out the smtp section of the /app/data/production.yaml file.

From /app/code/server I was then able to run /app/data/start.sh and the server started up properly:

root@d2fb1510-462d-45d0-84b8-8143184152fc:/app/code/server# /app/data/start.sh 
==> Updating Ldap credentials
UPDATE 0
==> Updating configs
==> Starting PeerTube

> peertube@2.4.0 start /app/code/server
> node dist/server

[peertube.cloudron.magnusonfinancial.com:443] 2020-12-10 14:02:20.940 warn: Emailer is disabled so the contact form will not work.
[peertube.cloudron.magnusonfinancial.com:443] 2020-12-10 14:02:21.319 info: Database dbd2fb1510462d45d084b88143184152fc is ready.
[peertube.cloudron.magnusonfinancial.com:443] 2020-12-10 14:02:21.455 error: Cannot use SMTP server because of lack of configuration. PeerTube will not be able to send mails!
[peertube.cloudron.magnusonfinancial.com:443] 2020-12-10 14:02:21.464 info: Cleaning HTML cache.
[peertube.cloudron.magnusonfinancial.com:443] 2020-12-10 14:02:21.507 info: Server listening on 0.0.0.0:9000

Based on this it seems like this app breaks if cloudron's outbound SMTP configuration is set to disabled.

@nebulon What do I do to report this as an issue and request the package be updated, is this message here enough or should I start a thread in the Support section of the forum, or something else?

nebulon

I think you went down the wrong path here. Unless the app is run in debug mode, it will always run /app/code/start.sh as the entry point. So any customization has to be done in /app/data/production.yaml for which we have to see if the kind of customization is even possible.

To take a step back, maybe you can describe what you would like to have changed and then we can see what is the best avenue to achieve that and if it is even possible.

jdaviescoates

@nebulon sounds like @ChristopherMag just wants to disable outbound email:

@christophermag said in Peertube restarting over and over:

When I initially installed peertube I had outbound email configured.
Due to a compliance issue I now have outbound email explicitly disabled so that no apps can send email external to the server.
I don't know if I restarted peertube since making that change so this issue may have been one waiting to show up the first time it restarted after having outbound email disabled.

nebulon

Ah true got it. Well the package is currently built in a way where the start.sh will always overwrite the settings, since secrets may or may not be rotated on the server side or port numbers change after first installation (While is is not done currently, we still have all packages done in a way they can be changed)

I don't really see an easy way to essentially "unconfigure" those without making the app error everytime it wants to send out a transactional email.

Ideally there would be some suppported "never send out emails" configure option, but this may be quite the edge-case. Maybe you can try to set the transport to either null or sendmail and see if the app still works?

ChristopherMag

@nebulon based on the current design I think the cleanest way to handle this is to change the behavior of Cloudron to be that when Outbound Email Relay configuration for a domain is set to disabled the smtp daemon referenced in the app containers via the environment variable CLOUDRON_MAIL_SMTP_SERVER continues to accept mail from applications and then deletes/routes all mail it receives to /dev/null (how to accomplish this would be dependent on the smtp daemon used).

This would allow individual apps not to have to have configuration changes based on whether email is enabled or not.

The other maybe more correct but difficult option is to change the state visible to the app container if Email Relay is set to disabled. This could be by no longer setting the various SMTP related environment variables so that the start.sh scripts can check for the presence of the variables before trying to use their values in configuration files.

The app state change option requires that each app's start.sh has to account for this configuration possibility and then change the app's configuration based on that whereas the first option allows all apps configuration to remain consistent independent of whether Email Relay is disabled or not.

girish

@christophermag Did I understand correctly that the issue is that peertube crashes if the SMTP server rejects the connection ? If so, I think I can make a patch for peertube upstream. Seems normal for SMTP servers to up/down, so an app should be able to cope with this anyways.

ChristopherMag

@girish Good thinking, yes that appears to be the behavior, if the smtp server rejects the connection the app refuses to start.

girish

@christophermag thanks, I can reproduce this easily.

girish

@christophermag I have opened an issue upstream and make a PR as well - https://github.com/Chocobozzz/PeerTube/issues/3457 and https://github.com/Chocobozzz/PeerTube/pull/3458

girish

@christophermag Change was merged, should be in next release of peertube.

ChristopherMag

@girish That is amazing, thank you!

Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.

Solved Peertube restarting over and over